人工智能数据

Data for AI

在许多流行的机器学习模型（包括 2012 年 AlexNet 使人工智能重新成为人们关注的焦点的非常成功的神经网络）的核心在于一个非常简单的数学问题：

At the core of many popular machine learning models, including the highly successful neural networks that brought artificial intelligence back into the popular spotlight with AlexNet in 2012, lies a very simple mathematical problem:

将给定的一组数据点拟合到适当的函数（将输入映射到输出）中，该函数拾取数据中的重要信号并忽略噪声，然后确保该函数在新数据上表现良好。

Fit a given set of data points into an appropriate function (mapping an input to an output) that picks up on the important signals in the data and ignores the noise, then make sure this function performs well on new data.

然而，复杂性和挑战来自多个方面：

Complexity and challenges, however, arise from various sources:

假设和特征

Hypothesis and features

生成数据的真实函数及其实际依赖的所有特征都是未知的。我们只是观察数据，然后尝试估计生成它的假设函数。我们的函数试图了解数据的哪些特征对于我们的预测、分类、决策或一般目的很重要。它还学习这些特征如何相互作用以产生观察到的结果。在这种情况下，人工智能的巨大潜力之一是它能够捕捉人类通常无法捕捉到的数据特征之间的微妙相互作用，因为我们非常擅长观察强大的特征，但可能会错过更微妙的特征。例如，我们作为人类可以知道一个人的月收入会影响他们偿还贷款的能力，但我们可能不会观察到他们的日常通勤或早上的例行公事也可能对此产生不小的影响。某些特征交互比其他特征交互简单得多，例如线性交互。其他的则更复杂并且是非线性的。从数学的角度来看，无论我们的特征交互是简单（线性）还是复杂（非线性），我们仍然有相同的目标：找到适合您的数据并能够对新数据做出良好预测的假设函数。这里出现了一个额外的复杂问题：有许多假设函数可以拟合相同的数据集——我们如何知道选择哪些函数？

Neither the true function that generated the data nor all the features it actually depends on are known. We simply observe the data, then try to estimate a hypothetical function that generated it. Our function tries to learn which features of the data are important for our predictions, classifications, decisions, or general purposes. It also learns how these features interact in order to produce the observed results. One of the great potentials of AI in this context is its ability to pick up on subtle interactions between features of data that humans do not usually pick up on, since we are very good at observing strong features but may miss more subtle ones. For example, we as humans can tell that a person’s monthly income affects their ability to pay back a loan, but we might not observe that their daily commute, or morning routine, may have a nontrivial effect on that as well. Some feature interactions are much simpler than others, such as linear interactions. Others are more complex and are nonlinear. From a mathematical point of view, whether our feature interactions are simple (linear) or complex (nonlinear), we still have the same goal: find the hypothetical function that fits your data and is able to make good predictions on new data. One extra complication arises here: there are many hypothetical functions that can fit the same data set—how do we know which ones to choose?

表现

Performance

即使在计算出适合我们数据的假设函数之后，我们如何知道它是否能够在新的和未见过的数据上表现良好？我们如何知道选择哪种性能衡量标准，以及如何在部署到现实世界后监控该性能？现实世界的数据和场景不会全部贴上基本事实标签，因此我们无法轻松衡量我们的人工智能系统是否表现良好并做出正确或适当的预测和决策。我们不知道用什么来衡量人工智能系统的结果。如果现实世界的数据和场景被贴上了基本事实的标签，那么我们都会破产，因为我们知道在每种情况下该做什么，地球上会有和平，我们会从此过上幸福的生活（不是真的，我希望事情就是这么简单）。

Even after computing a hypothetical function that fits our data, how do we know whether it will perform well on new and unseen data? How do we know which performance measure to choose, and how to monitor this performance after deploying into the real world? Real-world data and scenarios do not come to us all labeled with ground truths, so we cannot easily measure whether our AI system is doing well and making correct or appropriate predictions and decisions. We do not know what to measure the AI system’s results against. If real-world data and scenarios were labeled with ground truths, then we would all be out of business since we would know what to do in every situation, there would be peace on Earth, and we would live happily ever after (not really, I wish it was that simple).

体积

Volume

AI领域几乎所有的东西都是非常高维的！数据实例、观察到的特征和要计算的未知权重的数量可能为数百万，所需的计算步骤为数十亿。对此类数据进行高效存储、传输、探索、预处理、结构化和计算成为中心目标。此外，探索所涉及的高维数学函数的景观是一项不平凡的努力。

Almost everything in the AI field is very high-dimensional! The number of data instances, observed features, and unknown weights to be computed could be in the millions, and the required computation steps in the billions. Efficient storage, transport, exploration, preprocessing, structuring, and computation on such volumes of data become center goals. In addition, exploring the landscapes of the involved high-dimensional mathematical functions is a nontrivial endeavor.

结构

Structure

现代世界创建的大多数数据都是非结构化的。它不是以易于查询的表的形式组织的，这些表包含诸如姓名、电话号码、性别、年龄、邮政编码、房价、收入水平等标记字段。非结构化数据无处不在：社交媒体上的帖子、用户活动、 Word 文档、PDF 文件、图像、音频和视频文件、协作软件数据、交通或地震或天气数据、GPS、军事行动、电子邮件、即时消息、移动聊天数据等等。其中一些示例（例如电子邮件数据）可以被视为半结构化，因为电子邮件带有包含电子邮件元数据的标题：发件人、收件人、日期、时间、主题、内容类型、垃圾邮件状态等。重要数据无法以数字格式提供，并且分散在多个非通信数据库中。这里的例子包括历史军事数据、博物馆档案和医院记录。目前，我们的世界和城市数字化以利用更多人工智能应用的势头强劲。总体而言，从结构化和标记数据中获取见解比从非结构化数据中更容易。挖掘非结构化数据需要创新技术，这些技术目前是数据科学、机器学习和人工智能领域的驱动力智力。

Most of the data created by the modern world is unstructured. It is not organized in easy-to-query tables that contain labeled fields such as names, phone numbers, genders, ages, zip codes, house prices, income levels, etc. Unstructured data is everywhere: posts on social media, user activity, word documents, PDF files, images, audio and video files, collaboration software data, traffic or seismic or weather data, GPS, military movement, emails, instant messenger, mobile chat data, and many others. Some of these examples, such as email data, can be considered semistructured, since emails come with headings that include the email’s metadata: From, To, Date, Time, Subject, Content-Type, Spam Status, etc. Moreover, large volumes of important data are not available in digital format and are fragmented over multiple and noncommunicating databases. Examples here include historical military data, museum archives, and hospital records. Presently, there is great momentum toward digitalizing our world and our cities to leverage more AI applications. Overall, it is easier to draw insights from structured and labeled data than from unstructured data. Mining unstructured data requires innovative techniques that are currently driving forces in the fields of data science, machine learning, and artificial intelligence.

真实数据与模拟数据

Real Data Versus Simulated Data

当我们处理数据时，了解真实数据和模拟数据之间的区别非常重要。两种类型的数据对于人类的发现和进步都极其有价值。

When we work with data, it is very important to know the difference between real data and simulated data. Both types of data are extremely valuable for human discovery and progress.

真实数据

Real data

这数据是通过现实世界的观察收集的，使用测量设备、传感器、调查、结构化形式，如医学问卷、望远镜、成像设备、网​​站、股票市场、对照实验等。由于不准确和失败，这些数据通常是不完美和有噪音的在测量方法和仪器方面。从数学上讲，我们不知道生成真实数据的确切函数或概率分布，但我们可以使用模型、理论和模拟对其进行假设。然后我们可以测试我们的模型，最后用它们来做出预测。

This data is collected through real-world observations, using measuring devices, sensors, surveys, structured forms like medical questionnaires, telescopes, imaging devices, websites, stock markets, controlled experiments, etc. This data is often imperfect and noisy due to inaccuracies and failures in measuring methods and instruments. Mathematically, we do not know the exact function or probability distribution that generated the real data, but we can hypothesize about it using models, theories, and simulations. We can then test our models, and finally use them to make predictions.

模拟数据

Simulated data

这是使用已知函数生成的数据或从已知概率分布中随机采样的数据。在这里，我们有已知的数学函数或模型，并将数值插入模型中以生成数据点。例子很多：偏微分方程的数值解在各种尺度上模拟各种自然现象，例如湍流、蛋白质折叠、热扩散、化学反应、行星运动、断裂材料、交通，甚至增强迪士尼电影动画，例如模拟莫阿娜中的自然水运动或冰雪奇缘中艾莎的头发运动。

This is data generated using a known function or randomly sampled from a known probability distribution. Here, we have our known mathematical function(s), or model, and we plug numerical values into the model to generate our data points. Examples are plentiful: numerical solutions of partial differential equations modeling all kinds of natural phenomena on all kinds of scales, such as turbulent flows, protein folding, heat diffusion, chemical reactions, planetary motion, fractured materials, traffic, and even and even enhancing Disney movie animations, such as simulating natural water movement in Moana or Elsa’s hair movement in Frozen.

在本章中，我们将介绍两个有关人体身高和体重数据的示例，以展示真实数据和模拟数据之间的差异。在第一个示例中，我们访问一个在线公共数据库，然后下载并探索两个包含真实个体身高和体重测量值的真实数据集。在第二个示例中，我们根据我们假设的函数模拟我们自己的身高和体重数据集：我们假设一个人的体重与其身高线性相关。这意味着当我们根据身高数据绘制体重数据时，我们期望看到直线或平坦的视觉图案。

In this chapter, we present two examples about human height and weight data to demonstrate the difference between real and simulated data. In the first example, we visit an online public database, then download and explore two real data sets containing measurements of the heights and weights of real individuals. In the second example, we simulate our own data set of heights and weights based on a function that we hypothesize: we assume that the weight of an individual depends linearly on their height. This means that when we plot the weight data against the height data, we expect to see a straight, or flat, visual pattern.

数学模型：线性与非线性

Mathematical Models: Linear Versus Nonlinear

线性依赖关系模型世界的平坦度，如一维直线、二维平面（称为planes），和更高维的超平面。对线性相关性进行建模的线性函数图永远是平坦的并且不会弯曲。每当你看到一个平面物体，比如桌子、一根杆、一个天花板，或者一堆围绕一条直线或平面挤在一起的数据点时，你就知道它们的代表函数是线性的。任何不平坦的东西都是非线性的，因此图形弯曲的函数是非线性的，并且聚集在弯曲曲线或曲面周围的数据点是由非线性函数生成的。

Linear dependencies model flatness in the world, like one-dimensional straight lines, two-dimensional flat surfaces (called planes), and higher-dimensional hyperplanes. The graph of a linear function, which models a linear dependency, is forever flat and does not bend. Every time you see a flat object, like a table, a rod, a ceiling, or a bunch of data points huddled together around a straight line or a flat surface, know that their representative function is linear. Anything that isn’t flat is nonlinear, so functions whose graphs bend are nonlinear, and data points that congregate around bending curves or surfaces are generated by nonlinear functions.

线性函数的公式表示函数输出对特征或变量的线性依赖性，很容易写下来。这些特征仅以自身形式出现在公式中，没有幂或根，并且不嵌入任何其他函数中，例如分数分母、正弦、余弦、指数、对数或其他微积分函数。它们只能相乘由标量（实数或复数，而不是向量或矩阵）计算，并相互相加或相减。例如，线性依赖于三个特征的函数$X_1,X_2$， 和$X_3$可以写成：

The formula for a linear function, representing a linear dependency of the function output on the features, or variables, is very easy to write down. The features appear in the formula as just themselves, with no powers or roots, and are not embedded in any other functions, such as denominators of fractions, sine, cosine, exponential, logarithmic, or other calculus functions. They can only be multiplied by scalars (real or complex numbers, not vectors or matrices), and added to or subtracted from each other. For example, a function that depends linearly on three features $x_1,x_2$, and $x_3$ can be written as:

其中参数${\omega }_0,{\omega }_1,{\omega }_2$， 和${\omega }_3$是标量。参数或权重 ${\omega }_1,{\omega }_2$， 和${\omega }_3$线性组合特征，并产生结果$F（X_1,X_2,X_3）$添加偏置项后${\omega }_0$。换句话说，结果是特征之间线性相互作用的结果$X_1,X_2$， 和$X_3$，加上偏差。

where the parameters ${\omega }_0,{\omega }_1,{\omega }_2$, and ${\omega }_3$ are scalar numbers. The parameters or weights ${\omega }_1,{\omega }_2$, and ${\omega }_3$ linearly combine the features, and produce the outcome of $f(x_1,x_2,x_3)$ after adding the bias term ${\omega }_0$. In other words, the outcome is produced as a result of linear interactions between the features $x_1,x_2$, and $x_3$, plus bias.

非线性函数的公式表示函数输出对特征的非线性依赖性，也很容易发现。函数公式中出现的一个或多个特征的幂不是一，或者乘以或除以其他特征，或者嵌入到一些其他微积分函数中，例如正弦、余弦、指数、对数等。以下是三个示例函数依赖于三个特征非线性$X_1,X_2$， 和$X_3$:

The formula for a nonlinear function, representing a nonlinear dependency of the function output on the features, is very easy to spot as well. One or more features appear in the function formula with a power other than one, or multiplied or divided by other features, or embedded in some other calculus functions, such as sines, cosines, exponentials, logarithms, etc. The following are three examples of functions depending nonlinearly on three features $x_1,x_2$, and $x_3$:

正如您所知，我们可以提出各种非线性函数，并且与我们可以做什么以及我们可以使用非线性相互作用对世界的多少进行建模相关的可能性是无限的。事实上，神经网络之所以成功，是因为它们能够识别数据特征之间的相关非线性相互作用。

As you can tell, we can come up with all kinds of nonlinear functions, and the possibilities related to what we can do and how much of the world we can model using nonlinear interactions are limitless. In fact, neural networks are successful because of their ability to pick up on the relevant nonlinear interactions between the data features.

我们将在整本书中使用以前的符号和术语，因此您将非常熟悉线性组合、权重、特征以及之间的线性和非线性相互作用等术语。特征。

We will use the previous notation and terminology throughout the book, so you will become very familiar with terms like linear combination, weights, features, and linear and nonlinear interactions between features.

真实数据的例子

An Example of Real Data

你可以在本书的 GitHub 页面找到用于研究数据并生成以下两个示例中的图形的 Python 代码。

You can find the Python code to investigate the data and produce the figures in the following two examples at the book’s GitHub page.

我从Kaggle数据科学家网站下载了两个数据集。这两个数据集都包含一定数量个体的身高、体重和性别信息。我的目标是了解一个人的体重如何取决于他们的身高。从数学上讲，我想写一个公式，将重量作为一个特征（身高）的函数：

I downloaded two data sets from the Kaggle website for data scientists. Both data sets contain height, weight, and gender information for a certain number of individuals. My goal is to learn how the weight of a person depends on their height. Mathematically, I want to write a formula for the weight as a function of one feature, the height:

这样，如果给我一个新人的身高，我就能够预测他们的体重。当然，除了身高之外，一个人的体重还取决于其他特征，比如性别、饮食习惯、锻炼习惯、遗传倾向等。但是，对于我下载的数据集，我们只有身高、体重和体重。可获得性别数据。除非我们想要寻找更详细的数据集，或者出去收集新数据，否则我们必须利用现有的数据。此外，这个例子的目的只是为了说明真实数据和模拟数据之间的差异。当我们有更多复杂的目标时，我们可以使用具有更多特征的更复杂的数据集。

so that if I am given the height of a new person, I would be able to predict their weight. Of course, there are other features than height that a person’s weight depends on, such as their gender, eating habits, workout habits, genetic predisposition, etc. However, for the data sets that I downloaded, we only have height, weight, and gender data available. Unless we want to look for more detailed data sets, or go out and collect new data, we have to work with what we have. Moreover, the goal of this example is only to illustrate the difference between real data and simulated data. We can work with more involved data sets with larger numbers of features when we have more involved goals.

对于第一个数据集，我将体重列与高度列绘制在图 2-1中，并获得了一些似乎根本没有模式的东西！

For the first data set, I plot the weight column against the height column in Figure 2-1, and obtain something that seems to have no pattern at all!

图 2-1。当绘制第一个数据集的体重与身高的关系图时，我们无法检测到模式。散点图顶部和右侧的图表显示了身高和体重数据各自的直方图和经验分布。

对于第二个数据集，我也做了同样的事情，可以直观地观察到图2-2中明显的线性依赖关系。数据点似乎聚集在一条直线周围！

For the second data set, I do the same, and I can visually observe an obvious linear dependency in Figure 2-2. The data points seem to congregate around a straight line!

那么发生了什么？为什么我的第一个真实数据集反映了人的身高和体重之间没有任何依赖性，但我的第二个真实数据集却反映了线性依赖性？我们需要更深入地研究数据。

So what is going on? Why does my first real data set reflect no dependency between the height and weight of a person whatsoever, but my second one reflects a linear dependency? We need to look deeper into the data.

这是处理真实数据的众多挑战之一。我们不知道什么函数生成了这些数据，也不知道为什么它看起来是这样的。我们进行调查、获得见解、检测模式（如果有），然后提出假设函数。然后我们测试我们的假设，如果根据我们精心设计的绩效衡量标准表现良好，我们会将其部署到现实世界中。我们使用部署的模型进行预测，直到新数据告诉我们我们的假设不再有效，在这种情况下，我们研究更新的数据并制定新的假设。只要我们的模型投入使用，这个过程和反馈循环就会持续下去。

This is one of the many challenges of working with real data. We do not know what function generated the data, and why it looks the way it looks. We investigate, gain insights, detect patterns, if any, and we propose a hypothesis function. Then we test our hypothesis, and if it performs well based on our measures of performance, which have to be thoughtfully crafted, we deploy it into the real world. We make predictions using our deployed model, until new data tells us that our hypothesis is no longer valid, in which case we investigate the updated data and formulate a new hypothesis. This process and feedback loop keeps going for as long as our models are in business.

图 2-2。当绘制第二个数据集的体重与身高的关系图时，我们观察到线性模式。请注意，体重数据的经验分布绘制在图的右侧，身高数据的经验分布绘制在图的顶部。两者似乎都有两个峰（双峰），表明存在混合分布。事实上，身高和体重数据集都可以使用两种正态分布的混合（称为高斯混合）来建模，代表混合男性和女性数据的基础分布。因此，如果我们单独绘制女性或男性亚群的数据，如图2-6所示，我们会​​观察到身高和体重数据呈正态分布（钟形）。

在继续讨论模拟数据之前，让我们解释一下为什么第一个数据集似乎根本没有了解个体身高和体重之间的关系。经过进一步检查，我们注意到数据集中指数得分为 4 和 5（指肥胖和极度肥胖）的个体比例过高。因此，我决定按指数分数分割数据，并绘制具有相似指数分数的所有个体的体重与身高的关系图。这一次，身高和体重之间的线性相关性在图 2-3中很明显，谜团得到了解决。这可能让人感觉我们在通过调节个人的指数分数来欺骗我们的线性性。以数据探索的名义，一切都是公平的游戏。

Before moving on to simulated data, let’s explain why the first data set seemed to have no insight at all about the relationship between the height and the weight of an individual. Upon further inspection, we notice that the data set has an overrepresentation of individuals with Index scores 4 and 5, referring to obesity and extreme obesity. So, I decided to split the data by Index score, and plot the weight against the height for all individuals with similar Index scores. This time around, a linear dependency between the height and the weight is evident in Figure 2-3, and the mystery is resolved. This might feel like we are cheating our way to linearity, by conditioning on individuals’ Index scores. All is fair game in the name of data exploration.

图 2-3。当绘制第一个数据集中具有相似指数得分的个体的体重与身高的关系时，我们观察到线性模式。该图显示了指数得分为 3 的个体的体重与身高。

现在我们可以安全地假设重量与身高线性相关：

Now we can safely go ahead and hypothesize that the weight depends linearly on the height:

当然，我们剩下的任务是为参数找到合适的值${\omega }_0$和${\omega }_1$。第 3 章教我们如何做到这一点。事实上，机器学习（包括深度学习）的大部分活动都是关于学习这些$\omega$来自数据。在我们非常简单的例子中，我们只有两个$\omega$需要学习，因为我们只有一个特征，即高度，并且在观察真实数据中的线性模式后，我们假设了线性依赖性。在接下来的几章中，我们将遇到一些具有数百万个数据的深度学习网络$\omega$需要学习，但我们会发现问题的数学结构实际上与我们将要学习的结构完全相同在第 3 章中学习。

Of course, we are left with the task of finding appropriate values for the parameters ${\omega }_0$ and ${\omega }_1$. Chapter 3 teaches us how to do exactly that. In fact, the bulk of activity in machine learning, including deep learning, is about learning these $\omega$’s from the data. In our very simple example, we only have two $\omega$’s to learn, since we only had one feature, the height, and we assumed linear dependency after observing a linear pattern in the real data. In the next few chapters, we will encounter some deep learning networks with millions of $\omega$’s to learn, yet we will see that the mathematical structure of the problem is in fact the same exact structure that we will learn in Chapter 3.

模拟数据示例

An Example of Simulated Data

在这个例子，我模拟了我自己的身高体重数据集。模拟我们自己的数据可以避免从网络、现实世界中搜索数据甚至建立实验室来获得受控测量的麻烦。当所需数据无法获得或获取成本非常昂贵时，这是非常有价值的。它还可以通过仅更改函数中的数字来帮助测试不同的场景，而不是创建新材料或构建实验室并运行新实验。模拟数据非常方便，因为我们需要的只是一个数学函数、一个概率分布（如果我们想要涉及随机性和/或噪声）以及一台计算机。

In this example, I simulate my own height-weight data set. Simulating our own data circumvents the trouble of searching for data from the web, the real world, or even building a lab to obtain controlled measurements. This is incredibly valuable when the required data is not available or very expensive to obtain. It also helps test different scenarios by only changing numbers in a function, as opposed to, say, creating new materials or building labs and running new experiments. Simulating data is so convenient because all we need is a mathematical function, a probability distribution if we want to involve randomness and/or noise, and a computer.

让我们再次假设身高和体重之间存在线性相关性，因此我们将使用的函数是：

Let’s again assume linear dependency between the height and the weight, so the function that we will use is:

让我们能够模拟数值$（He我GHt,we我GHt）$对或数据点，我们必须假设参数的数值${\omega }_0$和${\omega }_1$。如果没有从真实数据中了解这些的正确选择$\omega$的，我们只能根据问题的背景做出有根据的猜测并尝试不同的值。请注意，对于本例中的身高体重情况，我们碰巧拥有真实数据，可以使用这些数据来学习适当的值$\omega$的，第 3 章的目标之一就是学习如何做到这一点。然而，在许多其他场景中，我们没有真实的数据，因此唯一的方法就是尝试这些数据的各种数值$\omega$的。

For us to be able to simulate numerical $(height,weight)$ pairs, or data points, we must assume numerical values for the parameters ${\omega }_0$ and ${\omega }_1$. Without having insights from real data about the correct choices for these $\omega$’s, we are left with making educated guesses from the context of the problem and experimenting with different values. Note that for the height-weight case in this example, we happen to have real data that we can use to learn appropriate values for the $\omega$’s, and one of the goals of Chapter 3 is to learn how to do that. However, in many other scenarios, we do not have real data, so the only way to go is experimenting with various numerical values for these $\omega$’s.

在下面的模拟中，我们设置${\omega }_0=-314。5$和${\omega }_1=7。07$，因此函数变为：

In the following simulations, we set ${\omega }_0=-314.5$ and ${\omega }_1=7.07$, so the function becomes:

现在我们可以生成尽可能多的数字$（He我GHt,we我GHt）$根据我们的需要配对。例如，堵漏$He我GHt=60$在权重函数的公式中，我们得到$we我GHt=-314。5+7。07\times 60=109。7$。所以我们的线性模型预测身高为 60 英寸的人的体重$109。7$磅，我们可以在身高体重图上绘制的数据点有坐标$（60,109。7）$。在图 2-4中，我们生成了 5,000 个这样的数据点：我们为 54 到 79 英寸之间的高度选择 5,000 个值，并将它们插入重量函数中。我们注意到图 2-4中的图表是一条完美的直线，模拟数据中没有噪声或变化，因为我们没有将这些纳入我们的线性模型中。

Now we can generate as many numerical $(height,weight)$ pairs as we want. For example, plugging $height=60$ in the formula for the weight function, we get $weight=-314.5+7.07\times 60=109.7$. So our linear model predicts that a person whose height is 60 inches weighs $109.7$ pounds, and the data point that we can plot on the height-weight graph has coordinates $(60,109.7)$. In Figure 2-4, we generate 5,000 of these data points: we choose 5,000 values for the height between 54 and 79 inches and plug them into the weight function. We notice that the graph in Figure 2-4 is a perfect straight line, with no noise or variation in the simulated data, since we did not incorporate those into our linear model.

这是模拟数据的一个标志：它执行生成它的函数执行的操作。如果我们理解用于构建模拟的函数（称为 model ），并且如果我们的计算没有积累太多数值错误和/或非常大的异常数字，那么我们就理解了模型生成的数据，并且我们可以以我们认为合适的任何方式使用这些数据。没有太多惊喜的空间。在我们的示例中，我们提出的函数是线性的，因此它的方程是一条直线，如图2-4所示，生成的数据完全位于这条直线上。

That’s a hallmark of simulated data: it does what the function(s) it was generated from does. If we understand the function (called model) that we used to build our simulation, and if our computation does not accumulate too many numerical errors and/or very large numbers that go rogue, then we understand the data that our model generates, and we can use this data in any way we see fit. There isn’t much space for surprises. In our example, our proposed function is linear, so its equation is that of a straight line, and as you see in Figure 2-4, the generated data lies perfectly on this straight line.

图 2-4。模拟数据：我们使用线性函数生成 5,000 个（身高、体重）点$we我GHt=-314。5+7。07\times He我GHt$

如果我们想模拟更真实的身高和体重数据怎么办？然后我们可以从更现实的人口身高分布中对身高值进行采样：钟形正态分布！同样，我们知道采样的概率分布，这与真实数据的情况不同。对高度值进行采样后，我们将它们插入到重量的线性模型中，然后添加一些噪声，因为我们希望模拟数据是真实的。由于噪声具有随机性，我们还必须选择从中采样的概率分布。我们再次选择钟形正态分布，但我们可以选择均匀分布来模拟均匀随机波动。我们更现实的身高体重模型变为：

What if we want to simulate more realistic data for height and weight? Then we can sample the height values from a more realistic distribution for the heights of a human population: the bell-shaped normal distribution! Again, we know the probability distribution that we are sampling from, which is different from the case for real data. After we sample the height values, we plug those into the linear model for the weight, then we add some noise, since we want our simulated data to be realistic. Since noise has a random nature, we must also pick the probability distribution it will be sampled from. We again choose the bell-shaped normal distribution, but we could have chosen the uniform distribution to model uniform random fluctuations. Our more realistic height-weight model becomes:

我们得到的结果如图 2-5所示。

We obtain the results seen in Figure 2-5.

图 2-5。模拟数据：我们使用线性函数生成 5,000 个（身高、体重）点$we我GHt=-314。5+7。07\times He我GHt$。高度点呈正态分布，我们还添加了正态分布的噪声。请注意，图中右侧和顶部的体重和身高数据的分布分别呈正态分布。这并不奇怪，因为我们在模拟中就是这样设计的。

现在将包含我们的模拟身高体重数据的图 2-5与包含我们使用的第二个数据集中的5,000 名女性的真实身高体重数据的图 2-6进行比较。还不错，因为只花了五分钟的代码编写就生成了这些数据，而不是收集真实的数据！如果我们花更多的时间来调整我们的价值观$\omega$的，以及我们添加的正态分布噪声的参数（平均值和标准差），我们将获得一个看起来更好的模拟数据集。然而，我们将把这个模拟留在这里，因为很快我们的全部重点将集中在学习我们假设的适当参数值上。楷模。

Now compare Figure 2-5 containing our simulated height-weight data to Figure 2-6 containing real height-weight data of 5,000 females from the second data set that we used. Not too bad, given that it only took five minutes of code writing to generate this data, as opposed to collecting real data! Had we spent more time tweaking the values of our $\omega$’s, and the parameters for the normally distributed noise that we added (mean and standard deviation), we would’ve obtained an even better-looking simulated data set. However, we will leave this simulation here since very soon our whole focus will be on learning the appropriate parameter values for our hypothesized models.

图 2-6。真实数据：第二个数据集中5,000 名女性的体重数据与身高数据相对应。请注意，图中右侧和顶部的女性体重和身高数据的分布分别呈正态分布。有关更多详细信息，请参阅本书的 GitHub 页面以获取更多详细信息。

数学模型：模拟和人工智能

Mathematical Models: Simulations and AI

我们可以始终调整我们的数学模型，使其更加现实。我们是设计师，所以我们可以决定这些模型的内容。通常的情况是，模型越模仿自然，其中包含的数学对象就越多。因此，在构建数学模型时，通常需要在更接近现实与模型的简单性和数学分析和计算的可访问性之间进行权衡。不同的设计师提出了不同的数学模型，有些模型比其他模型更好地捕捉某些现象。随着捕捉自然行为的不断探索，这些模型不断改进和发展。值得庆幸的是，我们的计算能力在过去几十年中得到了显着提高，使我们能够创建和测试更复杂、更现实的数学模型。

We can always adjust our mathematical models to make them more realistic. We are the designers, so we get to decide what goes into these models. It is often the case that the more a model mimics nature, the more mathematical objects get incorporated within it. Therefore, while building a mathematical model, the usual trade-off is between getting closer to reality, and the model’s simplicity and accessibility for mathematical analysis and computation. Different designers come up with different mathematical models, and some capture certain phenomena better than others. These models keep improving and evolving as the quest to capture natural behaviors continues. Thankfully, our computational capabilities have dramatically improved in the past decades, enabling us to create and test more involved and realistic mathematical models.

大自然既非常细致又极其广阔。自然界中的相互作用范围从亚原子量子领域一直到星系际尺度。作为人类，我们一直在努力了解自然并捕捉其复杂的组成部分及其众多的相互联系和相互作用。我们这样做的原因是多种多样的。它们包括对生命和宇宙起源的纯粹好奇，创造新技术，增强通信系统，设计药物和发现疾病治疗方法，建造武器和防御系统，以及前往遥远的星球并可能在未来居住。数学模型提供了一种出色且几乎神奇的方式来描述自然及其所有细节，仅使用数字、函数、方程，并在面临不确定性时通过概率调用量化随机性。这些数学模型的计算机模拟使我们能够研究和可视化建模系统或现象的各种简单和复杂的行为。反过来，计算机模拟的见解除了提供更深入的数学见解之外，还有助于模型增强和设计。这种令人难以置信的正反馈循环使数学建模和模拟成为不可或缺的工具，随着我们计算能力的增强，这种工具得到了极大的增强。

Nature is at the same time very finely detailed and enormously vast. Interactions in nature range from the subatomic quantum realm all the way to the intergalactic scale. We, as humans, are forever trying to understand nature and capture its intricate components with their numerous interconnections and interplays. Our reasons for this are varied. They include pure curiosity about the origins of life and the universe, creating new technologies, enhancing communication systems, designing drugs and discovering cures for diseases, building weapons and defense systems, and traveling to distant planets and perhaps inhabiting them in the future. Mathematical models provide an excellent and almost miraculous way to describe nature with all its details using only numbers, functions, equations, and invoking quantified randomness through probability when faced with uncertainty. Computer simulations of these mathematical models enable us to investigate and visualize various simple and complex behaviors of the modeled systems or phenomena. In turn, insights from computer simulations aid in model enhancement and design, in addition to supplying deeper mathematical insights. This incredibly positive feedback cycle makes mathematical modeling and simulations an indispensable tool that is enhanced greatly with our increased computational power.

宇宙的各种现象可以用数学的抽象语言精确建模，这是宇宙的奥秘；人类能够发现和理解数学，并建造出对各种有用的强大技术装置，这是人类思维的奇迹。应用程序。同样令人印象深刻的是，这些设备的核心只是计算或传输数学，更具体地说，是一堆零和一。

It is a mystery of the universe that its various phenomena can be accurately modeled using the abstract language of mathematics, and it is a marvel of the human mind that it can discover and comprehend mathematics, and build powerful technological devices that are useful for all kinds of applications. Equally impressive is that these devices, at their core, are doing nothing but computing or transmitting mathematics, more specifically, a bunch of zeros and ones.

人类能够概括他们对简单数字的理解，一直到为各种尺度的自然现象建立和应用数学模型，这是一个引人注目的例子所学知识的概括，是人类智力的标志。在人工智能领域，通用人工智能（类人人工智能和超级人工智能）和狭义人工智能（面向特定任务）的共同目标是泛化：人工智能代理将学到的能力泛化到新的和未知的情况的能力。在第 3 章中，我们将了解狭义且面向任务的人工智能的这一原理：人工智能代理从数据中学习，然后对新的和未见过的数据产生良好的预测。

The fact that humans are able to generalize their understanding of simple numbers all the way to building and applying mathematical models for natural phenomena at all kinds of scales is a spectacular example of generalization of learned knowledge, and is a hallmark of human intelligence. In the AI field, a common goal for both general AI (human-like and super-AI) and narrow AI (specific task oriented) is generalization: the ability of an AI agent to generalize learned abilities to new and unknown situations. In Chapter 3, we will understand this principle for narrow and task-oriented AI: an AI agent learns from data, then produces good predictions for new and unseen data.

人工智能通过三种方式与数学模型和模拟进行交互：

AI interacts in three ways with mathematical models and simulations:

数学模型和模拟为人工智能系统创建数据进行训练。

Mathematical models and simulations create data for AI systems to train on.

自动驾驶汽车被一些人认为是人工智能的基准。在汽车的人工智能系统了解到这些是必须避免的不利事件之前，让智能汽车原型车冲下悬崖、撞到行人或撞入新的工作区域将会带来不便。模拟数据训练在这里特别有价值，因为模拟可以创建各种危险的虚拟场景，供汽车在上路之前进行训练。同样，模拟数据对于训练火星漫游器的人工智能系统、药物发现、材料设计、天气预报、航空、军事训练等也非常有帮助。

Self-driving cars are considered by some to be a benchmark for AI. It will be inconvenient to let intelligent car prototypes drive off cliffs, hit pedestrians, or crash into new work zones before the car’s AI system learns that these are unfavorable events that must be avoided. Training on simulated data is especially valuable here, as simulations can create all kinds of hazardous virtual situations for a car to train on before releasing it out on the roads. Similarly, simulated data is tremendously helpful for training AI systems for rovers on Mars, drug discovery, materials design, weather forecasting, aviation, military training, and so on.

人工智能增强了现有的数学模型和模拟。

AI enhances existing mathematical models and simulations.

人工智能具有巨大的潜力，可以帮助解决传统上数学模型和模拟困难和限制的领域，例如在离散化方程时学习模型中涉及的参数的适当值、适当的概率分布、网格形状和大小（精细的网格捕获精细的数据）不同空间和时间尺度上的细节和微妙行为），并将计算方法扩展到更长的时间或具有复杂形状的更大的域。导航、航空、金融、材料科学、流体动力学、运筹学、分子和核科学、大气和海洋科学、天体物理学、物理和网络安全等领域严重依赖数学建模和模拟。将人工智能功能集成到这些领域已经开始取得非常积极的成果。我们将在本书后面的章节中遇到人工智能增强模拟的示例。

AI has great potential to assist in areas that have traditionally been difficult and limiting for mathematical models and simulations, such as learning appropriate values for the parameters involved in the models, appropriate probability distributions, mesh shapes and sizes when discretizing equations (fine meshes capture fine details and delicate behaviors at various spatial and time scales), and scaling computational methods to longer times or to larger domains with complicated shapes. Fields like navigation, aviation, finance, materials science, fluid dynamics, operations research, molecular and nuclear sciences, atmospheric and ocean sciences, astrophysics, physical and cyber security, and many others rely heavily on mathematical modeling and simulations. Integrating AI capabilities into these domains is starting to take place with very positive outcomes. We will come across examples of AI enhancing simulations in later chapters of this book.

AI本身就是一个数学模型和模拟。

AI itself is a mathematical model and simulation.

人工智能的一大愿望是通过计算复制人类智能。成功的机器学习系统，包括具有所有架构和变体的神经网络，都是数学模型，旨在模拟人类与智能相关的任务，例如视觉、模式识别和泛化、通过自然语言进行通信以及逻辑推理。理解、情感体验、同理心和协作也与智力相关，为人类的成功和统治做出了巨大贡献，因此，如果我们想要实现通用人工智能，同时更深入地了解自然，我们还必须找到复制它们的方法智力和人脑的运作。这些领域的努力已经在进行中。我们要记住的是，在所有这些领域，机器所做的都是计算。机器计算自然语言处理的文档含义，组合和计算计算机视觉的数字图像像素，将音频信号转换为数字向量并计算人机交互的新音频等等。这样就很容易看出软件人工智能是一个大的数学模型和模拟。随着我们的进步，这一点将变得更加明显这本书。

One of the big aspirations of AI is to computationally replicate human intelligence. Successful machine learning systems, including neural networks with all their architectures and variations, are mathematical models aimed at simulating tasks that humans associate with intelligence, such as vision, pattern recognition and generalization, communication through natural language, and logical reasoning. Understanding, emotional experience, empathy, and collaboration are also associated with intelligence and have contributed tremendously to the success and domination of humankind, so we must also find ways to replicate them if we want to achieve general AI while gaining a deeper understanding of the nature of intelligence and the workings of the human brain. Efforts in these areas are already on the way. What we want to keep in mind is that in all these areas, what machines are doing is computing. Machines compute meanings of documents for natural language processing, combine and compute digital image pixels for computer vision, convert audio signals to vectors of numbers and compute new audio for human-machine interaction, and so on. It is then easy to see how software AI is one big mathematical model and simulation. This will become more evident as we progress in this book.

我们从哪里获取数据？

Where Do We Get Our Data From?

当我第一次决定进入人工智能领域时，我想运用我的数学知识来帮助解决我感兴趣的现实问题。我在一个饱受战争蹂躏的国家长大，目睹了许多问题的爆发、破坏，然后最终消散或得到解决，要么通过直接修复，要么通过围绕问题进行调整的人类网络并适应全新的（不稳定的）平衡。战争中的常见问题是不同供应链突然大规模中断、大部分电网突然遭到破坏、某些桥梁的定点轰炸导致整个道路网络突然瘫痪、恐怖分子网络突然出现、黑市、贩运、通货膨胀、和贫困。在这些情况下，数学可以帮助解决的问题数量是无限的，包括战争战术和战略。为了美国的安全，我的博士学位。在数学专业和大学任期内，我开始接触公司、政府机构和军队，寻找具有真实数据的真实项目。我提出免费帮助他们找到问题的解决方案。我不知道的是，并通过惨痛的教训才了解到，获取真实数据是最大的障碍。存在许多法规、隐私问题、机构审查委员会和其他障碍。即使在跨越了所有这些障碍之后，公司、机构和组织仍然倾向于保留他们的数据，即使他们知道自己没有充分利用这些数据，而且人们几乎不得不乞求才能获得真实的数据。事实证明，我的经历并不是独一无二的。同样的情况也发生在该领域的许多其他人身上。

When I first decided to enter the AI field, I wanted to apply my mathematical knowledge to help solve real-world problems that I felt passionate about. I grew up in a war-torn country, and saw many problems erupt, disrupt, then eventually dissipate or get resolved, either by direct fixes or by the human network adjusting around them and settling into completely new (unstable) equilibria. Common problems in war were sudden and massive disruptions to different supply chains, sudden destruction of large parts of the power grid, sudden paralysis of entire road networks by targeted bombing of certain bridges, sudden emergence of terrorist networks, black markets, trafficking, inflation, and poverty. The number of problems that math can help solve in these scenarios, including war tactics and strategy, is limitless. From the safety of the United States, my Ph.D. in mathematics, and my tenure at a university, I started approaching companies, government agencies, and the military, looking for real projects with real data to work on. I offered to help find solutions to their problems for free. What I did not know, and learned the hard way, was that getting real data was the biggest hurdle. There are many regulations, privacy issues, institutional review boards, and other obstacles standing in the way. Even after jumping through all these hoops, the companies, institutions, and organizations tend to hold on to their data, even when they know they are not making the best use of it, and one almost has to beg in order to get real data. It turned out that the experience I had was not unique. The same had happened to many others in the field.

这个故事并不是要阻止您获取训练人工智能系统所需的真实数据。重点是，如果您遇到所需数据所有者的犹豫和抵制，不要感到惊讶和沮丧。不断询问，就会有人愿意迈出这一信念的一步。

This story is not meant to discourage you from getting the real data that you need to train your AI systems. The point is to not get surprised and disheartened if you encounter hesitation and resistance from the owners of the data that you need. Keep asking, and someone will be willing to take that one leap of faith.

有时，您需要的数据可以在网络上公开获得。对于本章中的简单模型，我使用的数据集来自Kaggle网站。还有其他很棒的公共数据存储库，我不会在这里列出，但是使用“最佳数据存储库”等关键字进行简单的 Google 搜索将返回出色的结果。一些存储库面向计算机视觉，其他存储库面向自然语言处理、音频生成和转录、科学研究等。

Sometimes the data you need is available publicly on the web. For the simple models in this chapter, I am using data sets from the Kaggle website. There are other great public data repositories, which I will not list here, but a simple Google search with keywords like “best data repositories” will return excellent results. Some repositories are geared toward computer vision, others toward natural language processing, audio generation and transcription, scientific research, and so on.

抓取网络获取数据很常见，但你必须遵守你所爬行的网站的规则。你还必须学习如何爬行（有人说数据科学家和统计学家的区别在于数据科学家知道如何破解！）。有些网站要求您在抓取之前获得书面许可。例如，如果您对社交媒体用户行为或协作网络感兴趣，您可以抓取社交媒体和专业网络（Facebook、Instagram、YouTube、Flickr、LinkedIn 等）以获取用户帐户的统计信息，例如数量这些网站上的朋友或联系、喜欢、评论和活动。您最终将得到包含数十万条记录的非常大的数据集，然后您可以对其进行计算。

Crawling the web to acquire data is common, but you have to abide by the rules of the websites you are crawling. You also have to learn how to crawl (some people say that the difference between data scientists and statisticians is that data scientists know how to hack!). Some websites require you to obtain written permission before you crawl. For example, if you are interested in social media user behavior, or in collaboration networks, you can crawl social media and professional networks—Facebook, Instagram, YouTube, Flickr, LinkedIn, etc.—for statistics on user accounts, such as the number of friends or connections, likes, comments, and activity on these sites. You will end up with very large data sets with hundreds of thousands of records, which you can then do your computations on.

为了直观地了解数据如何集成到人工智能中，以及进入各种系统的数据类型，同时避免被所有信息和数据淹没的感觉，开发一个探索成功的人工智能系统所训练的数据集（如果有）的习惯。您不必下载它们并进行操作。浏览数据集、其元数据、其附带的功能和标签（如果有）等，足以让您熟悉数据。例如，我们将在第 7 章中了解DeepMind 的WaveNet (2016) ，它是一个神经网络，可以生成原始机器音频，具有逼真的人声或令人愉悦的音乐片段。它可以完成诸如文本到音频转换之类的任务，并具有自然的人声含义，即使是特定人的声音（如果网络以该人的声音为条件）也是如此。我们将理解数学意义当我们在第 7 章学习 WaveNet 时进行调节。目前，将其视为对问题人为施加的限制，因此它将其结果限制为一组特定的结果。那么 WaveNet 是用什么数据训练的呢？对于不以文本为条件的多说话者音频生成，WaveNet 在音频文件数据集上进行了训练，该数据集包含来自 109 个不同说话者的 44 小时音频：来自 CSTR 语音克隆工具包 (2012) 的英语多说话者语料库。为了将文本转换为语音，WaveNet 在北美英语数据集（包含 24 小时的语音数据）和中文普通话数据集（包含 34.8 小时的语音数据）上进行了训练。为了生成音乐，WaveNet 在YouTube Piano 数据集上进行了训练，该数据集包含从 YouTube 视频中获得的 60 小时的钢琴独奏音乐，以及MagnaTagATune 数据集(2009)，该数据集包含约 200 小时的音乐音频，其中每个 29-第二个剪辑标有 188 个标签，描述音乐的流派、乐器、节奏、音量、情绪和各种其他标签。标记数据对于人工智能系统非常有价值，因为它提供了衡量假设函数输出的基本事实。我们将在接下来的几年中学习这一点部分。

To gain an intuitive understanding of how data gets integrated into AI, and the type of data that goes into various systems, while at the same time avoiding feeling overwhelmed by all the information and the data that is out there, it is beneficial to develop a habit of exploring the data sets that successful AI systems were trained on, if they are available. You do not have to download them and work on them. Browsing the data set, its metadata, what features and labels (if any) it comes with, etc., is enough to get you comfortable with data. For example, DeepMind’s WaveNet (2016), which we will learn about in Chapter 7, is a neural network that generates raw machine audio, with realistic-sounding human voices or enjoyable pieces of music. It accomplishes tasks like text-to-audio conversion with natural-sounding human voice connotations, even with a specific person’s voice if the network gets conditioned on this person’s voice. We will understand the mathematical meaning of conditioning when we study WaveNet in Chapter 7. For now, think of it as a restriction imposed artificially on a problem so it restricts its results to a certain set of outcomes. So what data was WaveNet trained on? For multispeaker audio generation that is not conditioned on text, WaveNet was trained on a data set of audio files consisting of 44 hours of audio from 109 different speakers: the English Multispeaker Corpus from CSTR Voice Cloning Toolkit (2012). For converting text to speech, WaveNet was trained on the North American English data set, which contains 24 hours of speech data, and on the Mandarin Chinese data set, which has 34.8 hours of speech data. For generating music, WaveNet was trained on the YouTube Piano data set, which has 60 hours of solo piano music obtained from YouTube videos, and the MagnaTagATune data set (2009), which consists of about 200 hours of music audio, where each 29-second clip is labeled with 188 tags describing the genre, instrumentation, tempo, volume, mood, and various other labels for the music. Labeled data is extremely valuable for AI systems, because it provides a ground truth to measure the output of your hypothesis function against. We will learn this in the next few sections.

著名的图像分类怎么样（用于计算机视觉）亚历克斯网络（2012）？其卷积神经网络使用哪些数据进行训练和测试？AlexNet 在ImageNet上进行训练，该数据集包含数百万张图像（从互联网上抓取）并用数千个类别进行标记（众包人类标签）。

How about the famous image classification (for computer vision) AlexNet (2012)? What data was its convolutional neural network trained and tested on? AlexNet was trained on ImageNet, a data set containing millions of images (scraped from the internet) and labeled (crowdsourced human labelers) with thousands of classes.

请注意，所有这些示例都是非结构化数据的示例。

Note that all of these examples were all examples of unstructured data.

如果某个系统训练所用的数据不是公开的，那么最好查找有关该系统的已发表论文或其文档，并了解如何获取所需数据。仅此一点就能教会你很多东西。

If the data that a certain system was trained on is not publicly available, it is good to look up the published paper on the system or its documentation and read about how the required data was obtained. That alone will teach you a lot.

在继续进行数学计算之前，请记住以下要点：

Before moving on to doing mathematics, keep in mind the following takeaways:

• 人工智能系统需要数字数据。

• AI systems need digital data.

• 有时，您需要的数据并不容易获取。

• Sometimes, the data you need is not easy to acquire.

• 有一场运动将我们的数字化全世界。

• There is a movement to digitalize our whole world.

数据分布、概率和统计词汇

The Vocabulary of Data Distributions, Probability, and Statistics

当你进入一个新领域，首先要学习的是该领域的词汇。这类似于学习一门新语言。你可以在教室里学习它，然后受苦，或者你可以去一个说这种语言的国家旅行，听一些常用术语。您不必知道“bonjour”在法语中的意思。但是当你在法国时，你注意到人们总是互相说这句话，所以你也开始这么说。有时您不会在正确的上下文中使用它，例如当您必须说“bonsoir”而不是“bonjour”时。但慢慢地，当您发现自己在法国停留的时间更长时，您将在正确的上下文中使用正确的词汇。

When you enter a new field, the first thing you want to learn is the vocabulary of that field. It is similar to learning a new language. You can learn it in a classroom, and suffer, or you can travel to a country that speaks the language, and listen to frequently used terms. You don’t have to know what “bonjour” means in French. But while you are in France, you notice that people say it to each other all the time, so you start saying it as well. Sometimes you will not use it in the right context, like when you have to say “bonsoir” instead of “bonjour.” But slowly, as you find yourself staying longer in France, you will be using the right vocabulary in the right context.

尽可能快地学习词汇而不必掌握任何细节的另一个好处是，不同的领域用不同的术语指代相同的概念，因为那里存在大量的词汇冲突。这最终成为混乱的一大根源，因此也是语言障碍的体现。当您学习该领域的常用词汇时，您会意识到您可能已经知道这些概念，只是现在您对它们有了新名称。

One more advantage of learning the vocabulary as fast as you can, without necessarily mastering any of the details, is that different fields refer to the same concepts with different terms, since there is a massive vocabulary collision out there. This ends up being a big source of confusion, therefore the embodiment of language barriers. When you learn the common vocabulary of the field, you will realize that you might already know the concepts, except that now you have new names for them.

为了人工智能应用的目的，你想知道的概率和统计词汇并不太多。一旦我们开始使用每个术语，我就会对其进行定义，但请注意，概率论的目标是对随机或随机数量和事件做出确定性陈述，因为人类讨厌不确定性，喜欢他们的世界是可控和可预测的。每当您阅读有关人工智能、机器学习或数据科学的内容时，请注意以下来自概率和统计领域的语言。再次强调，您还不必了解任何定义；只需了解其中的任何定义即可。您只需要听到以下各节中讨论的术语并熟悉它们各自的进展方式即可。

The vocabulary terms from probability and statistics that you want to know for the purposes of AI applications are not too many. I will define each term once we get to use it, but note that the goal of probability theory is to make deterministic statements about random or stochastic quantities and events, since humans hate uncertainty and like their world to be controllable and predictable. Watch for the following language from the fields of probability and statistics whenever you are reading about AI, machine learning, or data science. Again, you do not have to know any of the definitions yet; you just need to hear the terms discussed in the following sections and be familiar with the way they progress after each other.

连续分布与离散分布（密度与质量）

Continuous Distributions Versus Discrete Distributions (Density Versus Mass)

什么时候我们处理连续分布，重要的是使用观察或采样某个值附近或周围的数据点等术语，而不是观察或采样精确值。事实上，在这种情况下观察到精确值的概率为零。

When we deal with continuous distributions, it is important to use terms like observing or sampling a data point near or around a certain value instead of observing or sampling an exact value. In fact, the probability of observing an exact value in this case is zero.

当我们的数字处于连续体时，一个值和下一个值之间不存在离散的分离。实数具有无限的精度。例如，如果我测量一个男性的身高，结果是 6 英尺，我不知道我的测量值到底是 6 英尺，还是 6.00000000785 英尺，还是 5.9999111134255 英尺。最好将我的观测设置在 6 英尺左右的区间内，例如 5.95 <高度< 6.05，然后量化观测到 5.95 到 6.05 英尺之间高度的概率。

When our numbers are in the continuum, there is no discrete separation between one value and the next value. Real numbers have an infinite precision. For example, if I measure the height of a male and I get 6 feet, I wouldn’t know whether my measurement is exactly 6 feet or 6.00000000785 feet or 5.9999111134255 feet. It’s better to set my observation in an interval around 6 feet, for example 5.95 < height < 6.05, then quantify the probability of observing a height between 5.95 and 6.05 feet.

对于离散随机变量，我们不必担心，因为我们可以轻松地将可能的值彼此分开。例如，当我们掷骰子时，可能的值为 1、2、3、4、5 或 6。因此，我们可以自信地断言掷出精确 5 的概率是 1/6。此外，离散随机变量可以产生非数值结果；例如，当我们抛硬币时，我们可能的值是正面或反面。连续随机变量只能有数值结果。

We do not have such a worry for discrete random variables, as we can easily separate the possible values from each other. For example, when we roll a die, our possible values are 1, 2, 3, 4, 5, or 6. So we can confidently assert that the probability of rolling an exact 5 is 1/6. Moreover, a discrete random variable can have nonnumerical outcomes; for example, when we flip a coin, our possible values are heads or tails. A continuous random variable can only have numerical outcomes.

由于这个推理，当我们有一个连续随机变量时，我们定义它的概率密度函数，而不是它的概率质量函数，就像离散随机变量的情况一样。密度指定了一定长度、面积或体积的空间内存在多少物质（取决于我们所处的维度）。为了找到指定区域中物质的质量，我们将密度乘以所考虑区域的长度、面积或体积。如果给定每个无限小区域的密度，那么我们必须对整个区域进行积分才能找到该区域内的质量，因为积分类似于无数个无限小区域的总和。

Because of that reasoning, when we have a continuous random variable, we define its probability density function, not its probability mass function, as in the case of discrete random variables. A density specifies how much of a substance is present within a certain length or area or volume of space (depending on the dimension we’re in). To find the mass of a substance in a specified region, we multiply the density by the length, area, or volume of the considered region. If we are given the density per an infinitesimally small region, then we must integrate over the whole region to find the mass within that region, because an integral is akin to a sum over infinitely many infinitesimally small regions.

我们将在第 11 章中详细阐述这些想法并用数学形式化它们。目前，我们强调以下几点：

We will elaborate on these ideas and mathematically formalize them in Chapter 11. For now, we stress the following:

• 如果我们只有一个连续随机变量，例如某一人群中男性的身高，那么我们用一维概率密度函数来表示其概率分布：$F（X_1）$。为了找到高度在 5.95 < height < 6.05 之间的概率，我们对概率密度函数进行积分$F（X_1）$在区间 (5.95, 6.05) 上，我们写：

  $磷（5。95<He我GHt<6。05）={\int }_{5。95}^{6。05}F（X_1）d{X}_1$

• If we only have one continuous random variable, such as the height of males in a certain population, then we use a one-dimensional probability density function to represent its probability distribution: $f\left(x_1\right)$. To find the probability of the height being between 5.95 < height < 6.05, we integrate the probability density function $f\left(x_1\right)$ over the interval (5.95, 6.05), and we write:

  $P(5.95<height<6.05)={\int }_{5.95}^{6.05}f\left(x_1\right)d{x}_1$

• 如果我们有两个连续随机变量，例如某个人群中男性的身高和体重，或者一个人的真实身高和测量身高（通常包含随机噪声），那么我们使用二维概率密度函数来表示它们的联合概率分布：$F（X_1,X_2）$。所以为了求出身高在 5.95 < height < 6.05之间、体重在 160 < Weight < 175 之间的联合概率，我们对联合概率密度函数进行双重积分$F（X_1,X_2）$，假设我们知道公式$F（X_1,X_2）$，在区间 (5.95, 6.05) 和 (160,175) 上，我们写：

  $磷（5。95<He我GHt<6。05,160<we我GHt<175）={\int }_{160}^{175}{\int }_{5。95}^{6。05}F（X_1,X_2）d{X}_{1}d{X}_2$

• If we have two continuous random variables, such as the height and weight of males in a certain population, or the true height and the measured height of a person (which usually includes random noise), then we use a two-dimensional probability density function to represent their joint probability distribution: $f(x_1,x_2)$. So in order to find the joint probability that the height is between 5.95 < height < 6.05 and the weight is between 160 < weight < 175, we double integrate the joint probability density function $f(x_1,x_2)$, assuming that we know the formula for $f(x_1,x_2)$, over the intervals (5.95, 6.05) and (160,175), and we write:

  $P(5.95<height<6.05,160<weight<175)={\int }_{160}^{175}{\int }_{5.95}^{6.05}f(x_1,x_2)d{x}_{1}d{x}_2$

• 如果我们有两个以上的连续随机变量，那么我们使用更高维的概率密度函数来表示它们的联合分布。例如，如果我们有某个人群中男性的身高、体重和血压，那么我们使用三维联合概率分布函数：$F（X_1,X_2,X_3）$。与之前解释的推理类似，找到第一个随机变量介于$A<X_1<乙$，第二个随机变量介于$C<X_2<d$，第三个随机变量介于$e<X_3<F$，我们三重积分区间(a,b)、(c,d)和(e,f)上的联合概率密度函数，我们写：

  $磷（A<X_1<乙,C<X_2<d,e<X_3<F）={\int }_e^F{\int }_C^d{\int }_A^{乙}F（X_1,X_2,X_3）d{X}_{1}d{X}_{2}d{X}_3$

• If we have more than two continuous random variables, then we use a higher-dimensional probability density function to represent their joint distribution. For example, if we have the height, weight, and blood pressure of males in a certain population, then we use a three-dimensional joint probability distribution function: $f(x_1,x_2,x_3)$. Similar to the reasoning explained previously, to find the joint probability that the first random variable is between $a<x_1<b$, the second random variable is between $c<x_2<d$, and the third random variable is between $e<x_3<f$, we triple integrate the joint probability density function over the intervals (a,b), (c,d), and (e,f), and we write:

  $P(a<x_1<b,c<x_2<d,e<x_3<f)={\int }_e^f{\int }_c^d{\int }_a^{b}f(x_1,x_2,x_3)d{x}_{1}d{x}_{2}d{x}_3$

即使在为连续随机变量定义了概率密度函数之后，我们也并非所有的担忧（事物会在数学上相加）都被消除。同样，罪魁祸首是实数的无限精度。如果我们允许所有集合都有概率，我们就会遇到悖论，因为我们可以构造不相交的集合（例如分形集合或通过变换有理数集合而表示的集合），其概率加起来超过一！人们必须承认，这些场景是病态的，必须由一个有足够时间的人精心构建；然而，它们确实存在并产生悖论。数学中的测度论介入并提供了一个数学框架，我们可以在其中使用概率密度函数而不会遇到悖论。它定义了零测度集（这些集在我们工作的空间中不占据任何体积），然后为我们提供了大量定理，使我们几乎可以在任何地方进行计算，也就是说，除了零测度集之外。事实证明这对于我们来说已经足够了应用程序。

Not all our worries (that things will add up mathematically) are eliminated even after defining the probability density function for a continuous random variable. Again, the culprit is the infinite precision of real numbers. If we allow all sets to have probability, we encounter paradoxes in the sense that we can construct disjoint sets (such as fractal-shaped sets or sets formulated by transforming the set of rational numbers) whose probabilities add up to more than one! One must admit that these sets are pathological and must be carefully constructed by a person who has plenty of time on their hands; however, they exist and they produce paradoxes. Measure theory in mathematics steps in and provides a mathematical framework where we can work with probability density functions without encountering paradoxes. It defines sets of measure zero (these occupy no volume in the space that we are working in), then gives us plenty of theorems that allow us to do our computations almost everywhere, that is, except on sets of measure zero. This turns out to be more than enough for our applications.

联合概率密度函数的威力

The Power of the Joint Probability Density Function

拥有获得许多随机变量的联合概率分布是一件强大但罕见的事情。原因是联合概率分布在其中编码了每个单独随机变量的概率分布（边际分布），以及我们在这些随机变量之间遇到的所有可能的共现（和条件概率）。这类似于从上方观看整个城镇，而不是在城镇内部仅观察两条或更多小巷之间的一个交叉路口。

Having access to the joint probability distribution of many random variables is a powerful but rare thing. The reason is that the joint probability distribution encodes within it the probability distribution of each separate random variable (marginal distributions), as well as all the possible co-occurrences (and the conditional probabilities) that we ever encounter between these random variables. This is akin to seeing a whole town from above, rather than being inside the town and observing only one intersection between two or more alleys.

如果随机变量是独立的，则联合分布只是它们各自分布的乘积。然而，当随机变量不独立时，例如一个人的身高和体重，或者一个人的观测身高（包括测量噪声）和一个人的真实身高（不包括噪声），访问联合分发在存储方面要困难得多且昂贵。因随机变量情况下的联合分布是不可分离的，因此我们不能只单独存储其每个部分。我们需要存储两个或多个变量之间每个共现的每个值。随着相关随机变量数量的增加，存储需求（以及计算或搜索空间）呈指数增长，这是臭名昭著的维度诅咒。

If the random variables are independent, then the joint distribution is simply the product of each of their individual distributions. However, when the random variables are not independent, such as the height and the weight of a person, or the observed height of a person (which includes measurement noise) and the true height of a person (which doesn’t include noise), accessing the joint distribution is much more difficult and expensive storage-wise. The joint distribution in the case of dependent random variables is not separable, so we cannot only store each of its parts alone. We need to store every value for every co-occurrence between the two or more variables. This exponential increase in storage requirements (and computations or search spaces) as you increase the number of dependent random variables is one embodiment of the infamous curse of dimensionality.

当我们对联合概率密度函数进行切片时，比如说$F（X_1,X_2）$，这意味着当我们将其中一个随机变量（或更高维度的多个）固定为精确值时，我们检索到的分布与我们通常感兴趣的后验概率分布（给定观测值的模型参数的概率分布）。例如，切片$F（X_1,X_2）$在$X_1=A$，我们得到$F（A,X_2）$，它恰好与概率分布成正比$F（X_2|X_1=A）$（见图2-7）。

When we slice a joint probability density function, say $f(x_1,x_2)$, meaning when we fix one of the random variables (or more in higher dimensions) to be an exact value, we retrieve a distribution proportional to the posterior probability distribution (probability distribution of the model parameters given the observations), which we are usually interested in. For example, slice through $f(x_1,x_2)$ at $x_1=a$, and we get $f(a,x_2)$, which happens to be proportional to the probability distribution $f\left(x_2\right|x_1=a)$ (see Figure 2-7).

图 2-7。对联合概率分布进行切片

同样，这是在我们知道联合概率分布的豪华情况下，否则，我们使用贝叶斯规则来获得相同的后验概率分布（使用先验分布和似然函数）。

Again, this is in the luxurious case that we know the joint probability distribution, otherwise, we use Bayes’ Rule to obtain the same posterior probability distribution (using the prior distribution and the likelihood function).

在一些人工智能应用中，人工智能系统通过使用概率乘积规则将联合概率分布分离为条件概率的乘积来学习联合概率分布。一旦它学习了联合分布，它就会从中采样以生成新​​的有趣的数据。DeepMind 的 WaveNet 在生成原始音频的过程中做到了这一点。

In some AI applications, the AI system learns the joint probability distribution by separating it into a product of conditional probabilities using the product rule for probabilities. Once it learns the joint distribution, it then samples from it to generate new and interesting data. DeepMind’s WaveNet does that in its process of generating raw audio.

接下来的部分介绍人工智能应用程序中最有用的概率分布。两种普遍存在的连续分布是均匀分布和正态分布（也称为高斯分布），所以我们从这里开始。请参阅Jupyter Notebook来重现数字等细节。

The next sections introduce the most useful probability distributions for AI applications. Two ubiquitous continuous distributions are the uniform distribution and the normal distribution (also known as the Gaussian distribution), so we start there. Refer to the Jupyter Notebook for reproducing figures and more details.

数据分布：均匀分布

Distribution of Data: The Uniform Distribution

到为了直观地理解均匀分布，让我们举一个非均匀分布的例子，我们在本章前面已经看到过。在我们真实的身高体重数据集中，我们不能使用均匀分布来对身高数据进行建模。我们也不能用它来对体重数据建模。原因是人类的身高和体重分布不均匀。在一般人群中，遇到身高在 7 英尺左右的人与遇到身高在 5 英尺 6 英寸左右的人的可能性并不相同。

To intuitively understand the uniform distribution, let’s give an example of a nonuniform distribution, which we have already seen earlier in this chapter. In our real height-weight data sets, we cannot use the uniform distribution to model the height data. We also cannot use it to model the weight data. The reason is that human heights and weights are not evenly distributed. In the general population, it is not equally likely to encounter a person with height around 7 feet as it is to encounter a person with height around 5 feet 6 inches.

均匀分布仅对均匀分布的数据进行建模。如果我们有一个区间$（X_{米我n},X_{米AX}）$包含之间连续体中的所有值$X_{米我n}$和$X_{米AX}$的数据，并且我们的数据在我们的区间内均匀分布，那么观察到我们区间中任何特定值附近的数据点的概率对于该区间中的所有值都是相同的。也就是说，如果我们的间隔是$（0,1）$，选择 0.2 附近的点的可能性与选择 0.75 附近的点的可能性相同。

The uniform distribution only models data that is evenly distributed. If we have an interval $(x_{min},x_{max})$ containing all the values in the continuum between $x_{min}$ and $x_{max}$ of our data, and our data is uniformly distributed over our interval, then the probability of observing a data point near any particular value in our interval is the same for all values in this interval. That is, if our interval is $(0,1)$, it is equally likely to pick a point near 0.2 as it is to pick a point near 0.75.

这因此，均匀分布的概率密度函数是常数。对于一个区间内的一个随机变量 x$（X_{米我n},X_{米AX}）$，连续均匀分布的概率密度函数的公式由下式给出：

The probability density function for the uniform distribution is therefore constant. For one random variable x over an interval $(x_{min},x_{max})$, the formula for the probability density function for the continuous uniform distribution is given by:

否则为零。

and zero otherwise.

让我们绘制一个区间内均匀分布的概率密度函数$（X_{米我n},X_{米AX}）$。图 2-8中的图表是一条直线段，因为均匀分布的数据（无论是真实数据还是模拟数据）似乎均匀分布在所考虑的整个区间内。该区间内的数据值没有比其他值更容易出现。

Let’s plot the probability density function for the uniform distribution over an interval $(x_{min},x_{max})$. The graph in Figure 2-8 is a straight segment because uniformly distributed data, whether real or simulated, appears evenly distributed across the entire interval under consideration. No data values within the interval are more favored to appear than others.

图 2-8。区间 [0,1] 上均匀分布的概率密度函数图

均匀分布在计算机模拟中对于从任何其他概率分布生成随机数非常有用。如果您查看 Python 使用的随机数生成器，您会发现底层某处使用了均匀分布算法。

The uniform distribution is extremely useful in computer simulations for generating random numbers from any other probability distribution. If you peek into the random number generators that Python uses, you would see the uniform distribution used somewhere in the underlying algorithms.

数据分布：钟形正态（高斯）分布

Distribution of Data: The Bell-Shaped Normal (Gaussian) Distribution

A更适合模拟人体身高数据（当仅限于一种性别时）的连续概率分布是钟形正态分布，也称为高斯分布。正态分布的样本倾向于聚集在分布峰值处的平均值附近，称为平均值_ $\mu$，然后随着我们远离平均值而对称地缩小。分布在缩小时与平均值的距离有多远由正态分布的第二个参数控制，称为标准差 $\sigma$。大约 68% 的数据落在平均值的 1 个标准差范围内，95% 的数据落在平均值的 2 个标准差范围内，大约 99.7% 的数据落在平均值的 3 个标准差范围内（图2-9） 。

A continuous probability distribution better suited to model human height data (when restricted to one gender) is the bell-shaped normal distribution, also called the Gaussian distribution. Samples from the normal distribution tend to congregate around an average value where the distribution peaks, called the mean $\mu$, then dwindle symmetrically as we get farther away from the mean. How far from the mean the distribution spreads out as it dwindles down is controlled by the second parameter of the normal distribution, called the standard deviation $\sigma$. About 68% of the data falls within one standard deviation of the mean, 95% of the data falls within two standard deviations of the mean, and about 99.7% of the data falls within three standard deviations of the mean (Figure 2-9).

图 2-9。带参数的钟形正态分布的概率密度函数图$\mu =0$和$\sigma =1$

当我们从正态分布中采样数据时，接近平均值的值更有可能被选取（或发生或被观察），并且值非常小（$\to -\mathrm{无穷大}$）或非常大（$\to \mathrm{无穷大}$）被选中的可能性较小。这种在平均值附近达到峰值并在分布的外边缘衰减的现象使该分布具有著名的钟形形状。请注意，还有其他钟形连续分布，但正态分布是最普遍的。它为这个当之无愧的声誉提供了一个简洁的数学理由，基于概率论中的一个重要定理，称为中心极限定理（CLT）。

Values near the mean are more likely to be picked (or to occur, or to be observed) when we are sampling data from the normal distribution, and values that are very small ($\to -\infty$) or very large ($\to \infty$) are less likely to be picked. This peaking near the mean value and decaying on the outer skirts of the distribution give this distribution its famous bell shape. Note that there are other bell-shaped continuous distributions out there, but the normal distribution is the most prevalent. It has a neat mathematical justification for this well-deserved fame, based on an important theorem in probability theory called the central limit theorem (CLT).

中心极限定理指出，许多具有相同分布（不一定是正态分布）的独立随机变量的平均值呈正态分布。这就解释了为什么正态分布在社会和自然界中随处可见。它对婴儿出生体重、学生年​​级分布、国家收入分布、血压测量分布等进行建模。有特殊的统计测试可以帮助我们确定是否可以使用正态分布对真实数据集进行建模。我们将在第 11 章后面详细阐述这些想法。

The central limit theorem states that the average of many independent random variables that all have the same distribution (not necessarily the normal distribution) is normally distributed. This explains why the normal distribution appears everywhere in society and nature. It models baby birth weights, student grade distributions, countries’ income distributions, distribution of blood pressure measurements, etc. There are special statistical tests that help us determine whether a real data set can be modeled using the normal distribution. We will expand on these ideas later in Chapter 11.

如果您碰巧发现自己处于不确定并且不知道应用程序应使用哪种分布的情况，那么正态分布通常是一个合理的选择。事实上，在所有具有相同方差的分布选择中，正态分布就是选择最大的不确定性，因此它实际上将最少量的先验知识编码到模型中。

If you happen to find yourself in a situation where you are uncertain and have no prior knowledge about which distribution to use for your application, the normal distribution is usually a reasonable choice. In fact, among all choices of distributions with the same variance, the normal distribution is the choice with maximum uncertainty, so it does in fact encode the least amount of prior knowledge into your model.

正态分布的概率密度函数公式为一个具有均值的随机变量x （单变量）$\mu$和标准差$\sigma$是：

The formula for the probability density function of the normal distribution for one random variable x (univariate) with mean $\mu$ and standard deviation $\sigma$ is:

及其图表$\mu =0$和$\sigma =1$绘制如图2-9所示。

and its graph for $\mu =0$ and $\sigma =1$ is plotted in Figure 2-9.

两个随机数正态分布的概率密度函数公式变量x和y（双变量）是：

The formula for the probability density function for the normal distribution of two random variables x and y (bivariate) is:

其图形如图2-10所示。

and its graph is plotted in Figure 2-10.

我们可以使用线性代数语言以更紧凑的表示法写出上面的二元公式：

We can write the above bivariate formula in more compact notation using the language of linear algebra:

在图 2-11中，我们从二元正态分布中采样了 6,000 个点。靠近中心的点更有可能被选取，而远离中心的点则不太可能被选取。如果我们只观察样本点而不知道它们是从哪个分布中采样的，那么这些线大致描绘了正态分布的轮廓线。

In Figure 2-11, we sample 6,000 points from the bivariate normal distribution. Points near the center are more likely to be picked, and points away from the center are less likely to be picked. The lines roughly trace the contour lines of the normal distribution, had we only observed the sample points without knowing which distribution they were sampled from.

图 2-10。钟形二元正态分布的概率密度函数图

图 2-11。从二元正态分布中采样 6,000 个点

让我们暂停一下，将二元正态分布的概率密度函数公式与单变量正态分布的概率密度函数公式进行比较：

Let’s pause and compare the formula of the probability density function for the bivariate normal distribution to the formula of the probability density function for the univariate normal distribution:

• 当只有一个随机变量时，我们只有一个均值$\mu$和一个标准差$\sigma$。

• When there is only one random variable, we only have one mean $\mu$ and one standard deviation $\sigma$.

• 当有两个随机变量时，我们有两种均值$\left(\begin{array}{c}{\mu }_1\\ {\mu }_2\end{array}\right)$和两个标准差$\left(\begin{array}{c}{\sigma }_1\\ {\sigma }_2\end{array}\right)$。产品${\sigma }^2$将被协方差矩阵替换$\Sigma =\left(\begin{array}{cc}{\sigma }_1^2& \rho {\sigma }_1{\sigma }_2\\ \rho {\sigma }_1{\sigma }_2& {\sigma }_2^2\end{array}\right)$及其决定因素。$\rho$是两个随机变量之间的相关性，它是随机变量的两个归一化版本的协方差。

• When there are two random variables, we have two means $\left(\begin{array}{c}{\mu }_1\\ {\mu }_2\end{array}\right)$ and two standard deviations $\left(\begin{array}{c}{\sigma }_1\\ {\sigma }_2\end{array}\right)$. The product ${\sigma }^2$ will be replaced by the covariance matrix $\Sigma =\left(\begin{array}{cc}{\sigma }_1^2& \rho {\sigma }_1{\sigma }_2\\ \rho {\sigma }_1{\sigma }_2& {\sigma }_2^2\end{array}\right)$ and its determinant. $\rho$ is the correlation between the two random variables, which is the covariance of the two normalized versions of the random variables.

二元正态分布的概率密度函数的相同精确公式可以推广到任何维度，其中我们有许多随机变量而不是只有两个随机变量。例如，如果我们有 100 个随机变量，代表数据集中的 100 个特征，则公式中的均值向量将包含 100 个条目，协方差矩阵的大小为$100\times 100$，对角线上每个随机变量的方差以及对角线上 4,950 对随机变量中每一对的协方差对角线。

The same exact formula for the probability density function of the bivariate normal distribution generalizes to any dimension, where we have many random variables instead of only two random variables. For example, if we have 100 random variables, representing 100 features in a data set, the mean vector in the formula will have 100 entries in it, and the covariance matrix will have the size $100\times 100$, with the variance of each random variable on the diagonal and the covariance of each of the 4,950 pairs of random variables off the diagonal.

数据分布：其他重要且常用的分布

Distribution of Data: Other Important and Commonly Used Distributions

几乎所有你在本章中不理解的内容都会在本书中多次重温，第 11 章专门讨论概率。当这些概念在各种有趣的环境中一次又一次出现时，它们将会得到强化。本章的目标是接触概率和统计词汇，并为人工智能应用中经常出现的重要思想提供一个指导图。我们还希望对接下来的章节有一个良好的概率直觉，而不必深入研究并无缘无故地拖延我们的进度。

Almost everything you did not understand in this chapter will be revisited many times throughout the book, and Chapter 11 focuses exclusively on probability. The concepts will get reinforced as they appear again and again in various interesting contexts. Our goal for this chapter is to get exposed to the vocabulary of probability and statistics, and have a guiding map for the important ideas that frequently appear in AI applications. We also want to acquire a good probabilistic intuition for the following chapters without having to take a deep dive and delay our progress for no necessary reason.

那里有很多概率分布。每个模型都模拟了不同类型的现实世界场景。均匀分布和正态分布非常常见，但我们还有其他在人工智能领域经常出现的重要分布。回想一下，我们的目标是对我们周围的世界进行建模，以便做出良好的设计、预测和/或决策。当我们的模型涉及随机性或当我们不确定结果时，概率分布可以帮助我们做出预测。

There are many probability distributions out there. Each models a different type of real-world scenario. The uniform and normal distributions are very common, but we have other important distributions that frequently appear in the AI field. Recall that our goal is to model the world around us in order to make good designs, predictions, and/or decisions. Probability distributions help us make predictions when our models involve randomness or when we are uncertain about our outcomes.

当我们研究分布时，一个令人沮丧的部分是，它们中的大多数都有奇怪的名称，这些名称对于给定的分布对哪种现象有用提供了零直觉。这使得我们要么花费额外的精力来记住这些名称，要么在口袋里放一张发行备忘单。我更喜欢保留一份备忘单。另一个令人沮丧的部分是，大多数教科书的例子都涉及掷硬币、掷骰子或从瓮中抽彩球。这让我们没有现实生活中的例子或动机来理解这个主题，因为我从来没有见过有人四处走动抛硬币并数正面或反面，除了《黑暗骑士》中的双面人（又名哈维·登特）（2008 年的一部非常好的电影）电影中，小丑（希斯·莱杰饰演）说了一些我最喜欢的关于随机性和机会的深刻言论，比如：“世界是残酷的。残酷世界中唯一的道德……就是机会。不带偏见。不带偏见。” 。 公平的。”）。我将在本书中尽可能多地修改这一点，在页数限制允许的范围内指出尽可能多的现实世界的例子。

When we study distributions, one frustrating part is that most of them have weird names that provide zero intuition about what kind of phenomena a given distribution would be useful for. This makes us either expend extra mental energy to memorize these names, or keep a distribution cheat sheet in our pocket. I prefer to keep a cheat sheet. Another frustrating part is that most textbook examples involve flipping a coin, rolling a die, or drawing colored balls from urns. This leaves us with no real-life examples or motivation to understand the subject, as I never met anyone walking around flipping coins and counting heads or tails, except for Two-Face (aka Harvey Dent) in The Dark Knight (a really good 2008 movie, where the Joker [played by Heath Ledger] says some of my favorite and profound statements about randomness and chance, like this one: “The world is cruel. And the only morality in a cruel world…​is chance. Unbiased. Unprejudiced. Fair.”). I will try to amend this as much as I can in this book, pointing to as many real-world examples as my page limit allows.

以下一些分布在数学上彼此相关，或者自然地遵循其他分布。我们将在第 10 章中探讨这些关系。现在，让我们命名一个流行的分布，说明它是离散的（预测我们关心的事物的计数）还是连续的（预测连续体中存在的数量，例如某件事发生之前需要经过的时间；小心，这不是小时数，因为小时数是离散的，而是时间段的长度），说明控制它的参数，并说明其对我们的 AI 应用程序有用的定义属性：

Some of the following distributions are mathematically related to each other, or follow naturally from others. We will explore these relationships in Chapter 10. For now, let’s name a popular distribution, state whether it is discrete (predicts a count of something that we care for) or continuous (predicts a quantity that exists in the continuum, such as the time needed to elapse before something happens; careful, this is not the number of hours, since the number of hours is discrete, but it is the length of the time period), state the parameters that control it, and state its defining properties that are useful for our AI applications:

二项分布

Binomial distribution

这是离散的。它表示多次独立重复一项实验时获得一定数量成功的概率。它的控制参数是n，我们执行的实验数量，以及p，预定义的成功概率。现实世界的例子包括预测临床试验中疫苗或新药出现副作用的患者数量、导致购买的广告点击数量以及每月违约的客户数量信用卡付款。当我们使用需要独立试验的概率分布对现实世界的示例进行建模时，这意味着即使现实世界的试验并非真正独立，我们也假设独立。指出我们模型的假设是一种良好的礼仪。

This is discrete. It represents the probability of obtaining a certain number of successes when repeating one experiment, independently, multiple times. Its controlling parameters are n, the number of experiments we perform, and p, the predefined probability of success. Real-world examples include predicting the number of patients that will develop side effects for a vaccine or a new medication in a clinical trial, the number of ad clicks that will result in a purchase, and the number of customers that will default on their monthly credit card payments. When we model examples from the real world using a probability distribution that requires independent trials, it means that we are assuming independence even if the real-world trials are not really independent. It is good etiquette to point our models’ assumptions.

泊松分布

Poisson distribution

这是离散的。它预测在给定时间内将发生的罕见事件的数量。这些事件是独立的或弱相关的，意味着事件的发生一次不会影响其在同一时间段内下一次发生的概率。它们也以已知且恒定的平均速率发生$\lambda$。因此，我们知道平均发生率，并且我们想要预测在特定时间段内会发生多少此类事件。泊松分布的控制参数是预定义的罕见事件率$\lambda$。现实世界的例子包括预测特定小时内出生的婴儿数量、人口中 98 岁以上的人数、特定时间段内从放射性系统释放的 α 粒子数量、发送的重复账单数量美国国税局 (IRS) 统计的数据、某一天销售的不太受欢迎的产品数量、本书一页包含的错别字数量、某一天某台机器生产的缺陷产品数量、特定时间进入商店的人数、保险公司需要在特定时间段内承保的车祸次数以及特定时间段内发生的地震次数。

This is discrete. It predicts the number of rare events that will occur in a given period of time. These events are independent or weakly dependent, meaning that the occurrence of the event once does not affect the probability of its next occurrence in the same time period. They also occur at a known and constant average rate $\lambda$. Thus, we know the average rate, and we want to predict how many of these events will happen during a certain time period. The Poisson distribution’s controlling parameter is the predefined rare event rate $\lambda$. Real-world examples include predicting the number of babies born in a given hour, the number of people in a population who age past 98, the number of alpha particles discharged from a radioactive system during a certain time period, the number of duplicate bills sent out by the IRS, the number of a not-too-popular product sold on a particular day, the number of typos that one page of this book contains, the number of defective items produced by a certain machine on a certain day, the number of people entering a store at a certain hour, the number of car crashes an insurance company needs to cover within a certain time period, and the number of earthquakes happening within a particular time period.

几何分布

Geometric distribution

这是离散的。它预测我们在进行独立试验时获得成功之前所需的试验次数，每个试验都有一个已知的成功概率p 。这里的控制参数显然是成功的概率p 。现实世界的例子包括估计一家公司在没有出现网络故障的情况下可以运行的周数、一台机器在生产出有缺陷的产品之前可以运行的小时数，或者在遇到反对某个产品的人之前我们需要采访的人数。我们想要通过的某些政治法案。同样，对于这些现实世界的例子，如果使用几何分布建模，我们可能会假设独立性，而实际上试验可能不是独立的。

This is discrete. It predicts the number of trials needed before we obtain a success when performing independent trials, each with a known probability p for success. The controlling parameter here is obviously the probability p for success. Real-world examples include estimating the number of weeks that a company can function without experiencing a network failure, the number of hours a machine can function before producing a defective item, or the number of people we need to interview before meeting someone who opposes a certain political bill that we want to pass. Again, for these real-world examples, we might be assuming independence if modeling using the geometric distribution, while in reality the trials might not be independent.

指数分布

Exponential distribution

这是连续的。如果我们碰巧知道某个事件以恒定速率发生$\lambda$，然后指数分布预测该事件发生之前的等待时间。它是无记忆的，从某种意义上说，属于这种指数分布的项目的剩余寿命也是指数的。控制参数是恒定速率$\lambda$。现实世界的例子包括我们必须等待地震发生的时间、直到某人拖欠贷款的时间、直到机器零件发生故障的时间，或者恐怖袭击发生之前的时间。这对于可靠性领域非常有用，其中计算了某个机器部件的可靠性，因此有 10 年保修等声明。

This is continuous. If we happen to know that a certain event occurs at a constant rate $\lambda$, then exponential distribution predicts the waiting time until this event occurs. It is memoryless, in the sense that the remaining lifetime of an item that belongs to this exponential distribution is also exponential. The controlling parameter is the constant rate $\lambda$. Real-world examples include the amount of time we have to wait until an earthquake occurs, the time until someone defaults on a loan, the time until a machine part fails, or the time before a terrorist attack strikes. This is very useful for the reliability field, where the reliability of a certain machine part is calculated, hence statements such as a 10-year guarantee, etc.

威布尔分布

Weibull distribution

这是连续的。它广泛应用于预测产品寿命的工程领域（10 年保修声明也适用于此）。在这里，一个产品由许多部件组成，如果其中任何一个部件出现故障，那么该产品就会停止工作。例如，如果电池出现故障，或者变速箱中的保险丝烧断，汽车将无法工作。在考虑了汽车的许多部件及其最薄弱的环节（假设我们不维护汽车并重置时钟）之后，威布尔分布为汽车停止工作之前的使用寿命提供了一个很好的近似值。它由三个参数控制：形状、规模和位置。指数分布是该分布的特例，因为指数分布具有恒定的事件发生率，但威布尔分布可以对随时间增加或减少的发生率进行建模。

This is continuous. It is widely used in engineering in the field of predicting product lifetimes (10-year warranty statements are appropriate here as well). Here, a product consists of many parts, and if any of its parts fail, then the product stops working. For example, a car will not work if the battery fails, or if a fuse in the gearbox burns out. A Weibull distribution provides a good approximation for the lifetime of a car before it stops working, after accounting for its many parts and their weakest link (assuming we are not maintaining the car and resetting the clock). It is controlled by three parameters: shape, scale, and location. The exponential distribution is a special case of this distribution, because the exponential distribution has a constant rate of event occurrence, but the Weibull distribution can model rates of occurrence that increase or decrease with time.

对数正态分布

Log-normal distribution

这是连续的。如果我们取此分布中提供的每个值的对数，我们就会得到正态分布的数据。这意味着一开始，您的数据可能不会显示为正态分布，但如果您尝试使用日志函数对其进行转换，您将看到正态分布的数据。当遇到均值低、方差大且 仅假设正值的偏斜数据时，这是一个很好的分布。就像当您对随机变量的许多独立样本进行平均时（使用中心极限定理）会出现正态分布一样，当您取许多正样本值的乘积时会出现对数正态分布。从数学上讲，这是由于对数函数的一个令人敬畏的属性：乘积的对数是对数的总和。这种分布由三个参数控制：形状、尺度和位置。现实世界的例子包括石油储备中的天然气量，以及一天结束时证券价格与前一天结束时价格的比率。

This is continuous. If we take the logarithms of each value provided in this distribution, we get normally distributed data. Meaning that in the beginning, your data might not appear normally distributed, but if you try transforming it using the log function, you will see normally distributed data. This is a good distribution to use when encountering skewed data with low mean value, large variance, and assuming only positive values. Just like the normal distribution appears when you average many independent samples of a random variable (using the central limit theorem), the log-normal distribution appears when you take the product of many positive sample values. Mathematically, this is due to an awesome property of log functions: the log of a product is a sum of the logs. This distribution is controlled by three parameters: shape, scale, and location. Real-world examples include the volume of gas in a petroleum reserve, and the ratio of the price of a security at the end of one day to its price at the end of the day before.

卡方分布

Chi-squared distribution

这是连续的。它是正态分布的独立随机变量的平方和的分布。您可能想知道为什么我们要关心正态分布随机变量的平方，然后将它们相加。答案是，这就是我们通常计算随机变量或数据样本方差的方式，我们的主要目标之一是控制方差以降低不确定性。与此分布相关的显着性检验有两种类型：拟合优度检验（衡量我们的期望与观察结果的偏差程度）以及数据特征检验的独立性和同质性。

This is continuous. It is a distribution for the sum of squares of normally distributed independent random variables. You might wonder why would we care about squaring normally distributed random variables, then adding them up. The answer is that this is how we usually compute the variance of a random variable or of a data sample, and one of our main goals is controlling the variance in order to lower our uncertainties. There are two types of significance tests associated with this distribution: the goodness of fit test, which measures how far off our expectation is from our observation, and the independence and homogeneity of data features test.

帕累托分布

Pareto distribution

这是连续的。它对于许多现实世界的应用程序都很有用，例如完成分配给超级计算机的工作的时间（想想机器学习计算）、特定人群的家庭收入水平、社交网络中的朋友数量以及文件互联网流量的大小。该分布仅由一个参数控制$\alpha$，并且它是重尾的（它的尾部比指数分布更重）。

This is continuous. It is useful for many real-world applications, such as the time to complete a job assigned to a supercomputer (think machine learning computations), the household income level in a certain population, the number of friends in a social network, and the file size of internet traffic. This distribution is controlled by only one parameter $\alpha$, and it is heavy tailed (its tail is heavier than the exponential distribution).

在继续之前，让我们先介绍一些其他发行版，不要担心任何细节。这些都或多或少与上述发行版相关：

Let’s throw in few other distributions before moving on, without fussing about any of the details. These are all more or less related to the aforementioned distributions:

学生的 t 分布

Student’s t-distribution

连续，与正态分布类似，但在样本量较小且总体方差未知时使用。

Continuous, similar to the normal distribution, but used when the sample size is small and the population variance is unknown.

贝塔分布

Beta distribution

连续，在给定间隔内生成随机值。

Continuous, produces random values in a given interval.

柯西分布

Cauchy distribution

连续的、病态的，因为它的均值和方差都没有定义，可以使用随机选择的角度的正切来获得。

Continuous, pathological because neither its mean nor its variance is defined, can be obtained using the tangents of randomly chosen angles.

伽马分布

Gamma distribution

连续，与发生n 个独立事件之前的等待时间有关，而不是像指数分布那样仅发生一个事件。

Continuous, has to do with the waiting time until n independent events occur, instead of only one event, as in the exponential distribution.

负二项分布

Negative binomial distribution

离散，与获得一定数量的成功所需的独立试验的数量有关。

Discrete, has to do with the number of independent trials needed to obtain a certain number of successes.

超几何分布

Hypergeometric distribution

离散，类似于二项式，但试验不是独立的。

Discrete, similar to the binomial but the trials are not independent.

负超几何分布

Negative hypergeometric distribution

离散，捕获在我们获得一定数量的成功之前所需的相关试验的数量。

Discrete, captures the number of dependent trials needed before we obtain a certain number of successes.

“分发”一词的各种用法

The Various Uses of the Word “Distribution”

你可能已经注意到，“分布”一词指的是许多不同（但相关）的概念，具体取决于上下文。对同一词的不一致使用可能会造成混乱，并立即让一些试图进入该领域的人感到厌烦。

You might have already noticed that the word distribution refers to many different (but related) concepts, depending on the context. This inconsistent use of the same word could be a source of confusion and an immediate turnoff for some people who are trying to enter the field.

让我们列出“分布”一词所指的不同概念，以便我们在给定上下文中轻松识别其含义：

Let’s list the different concepts that the word distribution refers to, so that we easily recognize its intended meaning in a given context:

• 如果您有真实的数据，例如本章中的身高体重数据，并绘制数据集的一个特征（例如身高）的直方图，那么您会得到高度数据的经验分布。您通常不知道整个人口身高的潜在概率密度函数（也称为分布），因为您拥有的真实数据只是该人口的样本。因此，您尝试使用概率论给出的概率分布来估计它或对其进行建模。对于身高和体重特征，当按性别分开时，高斯分布是合适的。

• If you have real data, such as the height-weight data in this chapter, and plot the histogram of one feature of your data set, such as the height, then you get the empirical distribution of the height data. You usually do not know the underlying probability density function of the height of the entire population, also called distribution, since the real data you have is only a sample of that population. So you try to estimate it, or model it, using the probability distributions given by probability theory. For the height and weight features, when separated by gender, a Gaussian distribution is appropriate.

• 如果您有一个离散随机变量，则“分布”一词可以指其概率质量函数或其累积分布函数（指定随机变量小于或等于某个值的概率，$F（X）=pr哦乙（X\le X）$）。

• If you have a discrete random variable, the word distribution could refer to either its probability mass function or its cumulative distribution function (which specifies the probability that the random variable is less than or equal to a certain value, $f\left(x\right)=prob(X\le x)$).

• 如果您有一个连续随机变量，则单词分布可以指其概率密度函数或其累积分布函数，其积分给出随机变量小于或等于某个值的概率。

• If you have a continuous random variable, the word distribution could refer to either its probability density function or its cumulative distribution function, whose integral gives the probability that the random variable is less than or equal to a certain value.

• 如果您有多个随机变量（离散、连续或两者的混合），则单词分布指的是它们的联合概率分布。

• If you have multiple random variables (discrete, continuous, or a mix of both), then the word distribution refers to their joint probability distribution.

一个共同的目标是在理想化的数学函数（例如具有适当分布的随机变量）与具有观察到的经验分布的真实观察到的数据或现象之间建立适当的对应关系。在处理真实数据时，可以使用随机变量对数据集的每个特征进行建模。因此，在某种程度上，数学随机变量及其相应的分布是我们测量或观察到的特征的理想化版本。

A common goal is to establish an appropriate correspondence between an idealized mathematical function, such as a random variable with an appropriate distribution, and real observed data or phenomena, with an observed empirical distribution. When working with real data, each feature of the data set can be modeled using a random variable. So in a way, a mathematical random variable with its corresponding distribution is an idealized version of our measured or observed feature.

最后，分布在人工智能应用中随处可见。在接下来的章节中我们会多次遇到它们，例如神经网络每一层权重的分布，以及各种机器学习所犯的噪声和错误的分布楷模。

Finally, distributions appear everywhere in AI applications. We will encounter them plenty of times in the next chapters, for example, the distribution of the weights at each layer of a neural network, and the distribution of the noise and errors committed by various machine learning models.

A/B 测试

A/B Testing

前离开本章，我们将绕道进入 A/B 测试的世界，也称为对比测试，或随机单盲或双盲试验。我们绕道而行是因为这对数据科学家来说很重要：无数公司依靠 A/B 测试的数据来提高参与度、收入和客户满意度。微软、亚马逊、LinkedIn、谷歌和其他公司每年都会进行数千次 A/B 测试。

Before leaving this chapter, we make a tiny detour into the world of A/B testing, also called split testing, or randomized single-blind or double-blind trials. We make this detour because this is a topic important for data scientists: countless companies rely on data from A/B tests to increase engagement, revenue, and customer satisfaction. Microsoft, Amazon, LinkedIn, Google, and others each conduct thousands of A/B tests annually.

A/B 测试的想法很简单：将总体分为两组。将您想要测试的内容的版本（新的网页设计、不同的字体大小、新药物、新的政治广告）推出给一组（测试组），并将另一组保留为对照组。比较两组之间的数据。

The idea of an A/B test is simple: split the population into two groups. Roll out a version of something you want to test (a new web page design, a different font size, a new medicine, a new political ad) to one group, the test group, and keep the other group as a control group. Compare the data between the two groups.

考试如果受试者不知道他们属于哪个组（有些人甚至根本不知道他们正在参加测试），但实验者知道，则为单盲。如果实验者和受试者都不知道他们正在与哪一组进行交互，则测试是双盲的。

The test is single blind if the subjects do not know which group they belong to (some do not even know that they are in a test at all), but the experimenters know. The test is double blind if neither the experimenters nor the subjects know which group they are interacting with.

总结与展望

Summary and Looking Ahead

在本章中，我们强调了数据是人工智能的核心这一事实。我们还澄清了通常容易混淆的概念之间的差异：结构化和非结构化数据、线性和非线性模型、真实和模拟数据、确定性函数和随机变量、离散和连续分布以及后验概率和似然函数。我们还提供了人工智能所需的概率和统计图，而无需深入研究任何细节，并且我们还介绍了最流行的概率分布。

In this chapter, we emphasized the fact that data is central to AI. We also clarified the differences between concepts that are usually a source of confusion: structured and unstructured data, linear and nonlinear models, real and simulated data, deterministic functions and random variables, discrete and continuous distributions, and posterior probabilities and likelihood functions. We also provided a map for the probability and statistics needed for AI without diving into any of the details, and we introduced the most popular probability distributions.

如果您发现自己迷失在一些新的概率概念中，您可能需要查阅本章提供的地图，看看该概念如何适应概率论的大局，最重要的是，它与人工智能有何关系。如果不知道特定的数学概念与人工智能有何关系，你就会拥有一些你知道如何打开的工具，但你不知道它的用途。

If you find yourself lost in some new probability concept, you might want to consult the map provided in this chapter and see how that concept fits within the big picture of probability theory, and most importantly, how it relates to AI. Without knowing how a particular mathematical concept relates to AI, you are left with having some tool that you know how to turn on, but you have no idea what it is used for.

我们还没有提到随机矩阵和高维概率。在这些领域中，概率论不断跟踪任何相关随机量的分布、期望和方差，与高度关注特征值和各种矩阵分解的线性代数相结合。这些领域对于人工智能应用中涉及的极高维数据非常重要。我们将在第 11 章概率中讨论它们。

We have not yet mentioned random matrices and high-dimensional probability. In these fields, probability theory, with its constant tracking of distributions, expectations, and variances of any relevant random quantities, merges with linear algebra, with its hyperfocus on eigenvalues and various matrix decompositions. These fields are very important for the extremely high-dimensional data that is involved in AI applications. We discuss them in Chapter 11 on probability.

在下一章中，我们学习如何将数据拟合到函数中，然后使用该函数进行预测和/或决策。从数学上讲，我们找到权重（$\omega$'s）描述了我们的数据特征之间的各种相互作用的强度。当我们描述所涉及的交互类型（拟合函数的公式，称为学习或训练函数）以及这些交互的强度（$\omega$'s），我们可以做出我们的预测。在人工智能中，用合适的权重值来表征拟合函数的这一概念可以成功地用于计算机视觉、自然语言处理、预测分析（如房价、维护时间等）和许多其他应用。

In the next chapter, we learn how to fit our data into a function, then use this function to make predictions and/or decisions. Mathematically, we find the weights (the $\omega$’s) that characterize the strengths of various interactions between the features of our data. When we characterize the involved types of interactions (the formula of the fitting function, called the learning or training function) and the strengths of these interactions (the values of the $\omega$’s), we can make our predictions. In AI, this one concept of characterizing the fitting function with its suitable weight values can be used successfully for computer vision, natural language processing, predictive analytics (like house prices, time until maintenance, etc.), and many other applications.

传统且非常有用的机器学习模型

Traditional and Very Useful Machine Learning Models

一切本章中使用的数据被标记为真实数据，我们模型的目标是预测新的（未见过的）和未标记数据的标签。这就是监督学习。

All the data used in this chapter is labeled with ground truths, and the goal of our models is to predict the labels of new (unseen) and unlabeled data. This is supervised learning.

在接下来的几节中，我们使用以下流行的机器学习模型将训练函数拟合到标记数据中。虽然您可能听说了很多有关人工智能最新和最伟大发展的信息，但在典型的业务环境中，您可能最好从这些更传统的模型开始：

In the next few sections, we fit training functions into our labeled data using the following popular machine learning models. While you may hear so much about the latest and greatest developments in AI, in a typical business setting you are probably better off starting with these more traditional models:

线性回归

Linear regression

预测一个数值。

Predict a numerical value.

逻辑回归

Logistic regression

分为两类（二元分类）。

Classify into two classes (binary classification).

Softmax回归

Softmax regression

分为多个类。

Classify into multiple classes.

支持向量机

Support vector machines

分为两类，或回归（预测数值）。

Classify into two classes, or regression (predict a numerical value).

决策树

Decision trees

分类为任意数量的类别，或回归（预测数值）。

Classify into any number of classes, or regression (predict a numerical value).

随机森林

Random forests

分类为任意数量的类别，或回归（预测数值）。

Classify into any number of classes, or regression (predict a numerical value).

模型组合

Ensembles of models

通过平均预测值、投票选出最受欢迎的类别或其他一些捆绑机制，将许多模型的结果捆绑在一起。

Bundle up the results of many models by averaging the prediction values, voting for the most popular class, or some other bundling mechanism.

k-均值聚类

k-means clustering

分类为任意数量的类，或回归

Classify into any number of classes, or regression

我们在相同的数据集上尝试多个模型来比较性能。在现实世界中，很少有任何模型在没有与许多其他模型进行比较的情况下得到部署。这是计算量大的人工智能行业的本质，也是为什么我们需要并行计算，它使我们能够同时训练多个模型（除了构建和改进其他模型结果的模型，例如堆叠的情况）；对于那些我们不能使用的并行计算）。

We try multiple models on the same data sets to compare performance. In the real world, it is rare that any model ever gets deployed without having been compared with many other models. This is the nature of the computation-heavy AI industry, and why we need parallel computing, which enables us to train multiple models at once (except for models that build and improve on the results of other models, like in the case of stacking; for those we cannot use parallel computing).

在我们深入研究任何机器学习模型之前，非常重要的是要注意，据一再报道，数据科学家和/或人工智能研究人员只有大约 5% 的时间花在训练机器学习模型上。在将数据输入机器学习模型之前，大部分时间消耗在获取数据、清理数据、组织数据、为数据创建适当的管道等方面。因此，机器学习只是生产过程中的一个步骤，一旦数据准备好训练模型，这一步就很容易了。我们将发现这些机器学习模型是如何工作的：我们需要的大部分数学知识都存在于这些模型中。人工智能研究人员一直在尝试增强机器学习模型并将其自动融入生产流程。因此，对我们来说，最终了解整个管道非常重要，从原始数据（包括其存储、硬件、查询协议等）到部署再到监控。学习机器学习只是一个更大、更有趣的故事的一部分。

Before we dive into any machine learning models, it is extremely important to note that it has been reported again and again that only about 5% of a data scientist’s time, and/or an AI researcher’s time, is spent on training machine learning models. The majority of the time is consumed by acquiring data, cleaning data, organizing data, creating appropriate pipelines for data, etc., before feeding the data into machine learning models. So machine learning is only one step in the production process, and it is an easy step once the data is ready to train the model. We will discover how these machine learning models work: most of the mathematics we need resides in these models. AI researchers are always trying to enhance machine learning models and automatically fit them into production pipelines. It is therefore important for us to eventually learn about the whole pipeline, from raw data (including its storage, hardware, query protocols, etc.) to deployment to monitoring. Learning machine learning is only one piece of a bigger and more interesting story.

我们必须开始因为回归的思想对于接下来的大多数人工智能模型和应用程序来说都是非常基础的。仅对于线性回归，我们才能使用分析方法找到最小化权重，直接根据训练数据集及其目标标签给出所需权重的明确公式。线性回归模型的简单性允许这种显式解析解。大多数其他模型没有这样的显式解，我们必须使用数值方法找到它们的极小值，其中梯度下降是非常流行的。

We must start with regression since the ideas of regression are so fundamental for most of the AI models and applications that will follow. Only for linear regression do we find our minimizing weights using an analytical method, giving an explicit formula for the desired weights directly in terms of the training data set and its target labels. The simplicity of the linear regression model allows for this explicit analytical solution. Most other models do not have such explicit solutions, and we have to find their minimizers using numerical methods, among which the gradient descent is extremely popular.

在回归和许多其他即将推出的模型中，包括接下来几章的神经网络，请注意建模过程中的以下进展：

In regression and many other upcoming models, including the neural networks of the next few chapters, watch for the following progression in the modeling process:

1. 训练功能

2. The training function

3. 损失函数

4. The loss function

5. 优化

6. Optimization

数值解与分析解

Numerical Solutions Versus Analytical Solutions

这是了解数学问题的数值解和解析解之间的差异非常重要。数学问题可以是任何问题，例如：

It is important to be aware of the differences between numerical solutions and analytical solutions of mathematical problems. A mathematical problem can be anything, such as:

• 求某个函数的极小值。

• Find the minimizer of some function.

• 在预算有限的情况下找到从目的地A到目的地B的最佳路线。

• Find the best way to go from destination A to destination B, with a constrained budget.

• 找到设计和查询数据仓库的最佳方法。

• Find the best way to design and query a data warehouse.

• 找到数学方程的解（其中包含数学内容的左侧等于包含数学内容的右侧）。这些方程可以是代数方程、常微分方程、偏微分方程、积分微分方程、方程组或任何类型的数学方程。他们的解决方案可以是静态的，也可以随时间变化。他们可以对物理、生物、社会经济或自然世界中的任何事物进行建模。

• Find the solution of a mathematical equation (where a lefthand side with math stuff equals a righthand side with math stuff). These equations could be algebraic equations, ordinary differential equations, partial differential equations, integro-differential equations, systems of equations, or any sort of mathematical equations. Their solutions could be static or evolving in time. They could model anything from the physical, biological, socioeconomic, or natural worlds.

这是词汇：

Here is the vocabulary:

数值

Numerical

和数字有关系

Has to do with numbers

分析型

Analytical

与分析有关

Has to do with analysis

根据经验，只要我们有足够的计算能力来模拟和计算这些解决方案，数值解决方案比解析解决方案更容易获得且更容易获得。我们通常需要做的就是离散一些连续空间和/或函数（将连续体变成一堆点），尽管有时以非常聪明的方式，并根据这些离散量评估函数。数值解的唯一问题是它们只是近似解。除非它们得到了对它们与真实解析解的距离有多远以及它们收敛到这些真实解的速度的估计的支持，而这反过来又需要数学背景和分析，否则数值解是不精确的。然而，它们确实提供了关于真正解决方案的非常有用的见解。在许多情况下，数值解是唯一可用的，如果不依赖于复杂问题的数值解，许多科学和工程领域根本不会取得进步。如果这些领域等待解析解和证明出现，或者换句话说，等待数学理论迎头赶上，那么它们的进展将会非常缓慢。

As a rule of thumb, numerical solutions are much easier to obtain and much more accessible than analytical solutions, provided that we have enough computational power to simulate and compute these solutions. All we usually need to do is discretize some continuous spaces and/or functions (change the continuum into a bunch of points), albeit sometimes in very clever ways, and evaluate functions on these discrete quantities. The only problem with numerical solutions is that they are only approximate solutions. Unless they are backed by estimates on how far off they are from the true analytical solutions and how fast they converge to these true solutions, which in turn requires mathematical background and analysis, numerical solutions are not exact. They do, however, provide incredibly useful insights about the true solutions. In many cases, numerical solutions are the only ones available, and many scientific and engineering fields would not have advanced at all had they not relied on numerical solutions of complex problems. If those fields waited for analytical solutions and proofs to happen, or, in other words, for mathematical theory to catch up, they would have had very slow progress.

另一方面，解析解是精确的、稳健的，并且有完整的数学理论支持。它们附有定理和证明。当分析解决方案可用时，它们的作用非常强大。然而，它们并不容易获得，有时甚至不可能获得，并且它们确实需要微积分、数学分析、代数、微分方程理论等领域的深厚知识和领域专业知识。然而，分析方法对于描述解的重要属性（即使没有显式解），指导数值技术，并提供基本事实来比较近似数值方法（幸运的是，当这些解析解可用时）。

Analytical solutions, on the other hand, are exact, robust, and have a whole mathematical theory backing them up. They come accompanied with theorems and proofs. When analytical solutions are available, they are very powerful. They are, however, not easily accessible, sometimes impossible to obtain, and they do require deep knowledge and domain expertise in fields such as calculus, mathematical analysis, algebra, theory of differential equations, etc. Analytical methods, however, are extremely valuable for describing important properties of solutions (even when explicit solutions are not available), guiding numerical techniques, and providing ground truths to compare approximate numerical methods against (in the lucky cases when these analytical solutions are available).

一些研究人员纯粹是分析和理论的，其他研究人员是纯粹数值和计算的，最好的存在位置是在交叉点附近的某个地方，在那里我们对数学的分析和数值方面有很好的理解问题。

Some researchers are purely analytical and theoretical, others are purely numerical and computational, and the best place to exist is somewhere near the intersection, where we have a decent understanding of the analytical and the numerical aspects of our mathematical problems.

回归：预测数值

Regression: Predict a Numerical Value

一个快速在Kaggle 网站上搜索回归数据集会返回许多优秀的数据集和相关笔记本。我随机选择了一个简单的鱼市数据集，我们将用它来解释即将到来的数学。我们的目标是建立一个模型，根据鱼的五种不同的长度测量值或特征（在数据集中标记为长度 1、长度 2、长度 3、高度和宽度）来预测鱼的重量（见图 3-2 ）。为了简单起见，我们选择不将分类特征“物种”合并到该模型中，尽管我们可以这样做（这会给我们带来更好的预测，因为鱼的类型可以很好地预测其重量）。如果我们选择包含物种特征，那么我们必须将其值转换为数值，使用one-hot 编码，顾名思义：根据鱼的类别（类型）为每条鱼分配一个由 1 和 0 组成的代码。我们的物种功能有七个类别：鲈鱼、鳊鱼、蟑螂、梭子鱼、胡瓜鱼、白鱼和白鱼。因此，如果我们的鱼是梭子鱼，我们会将其物种编码为 (0,0,0,1,0,0,0)，如果它是鳊鱼，我们会将其物种编码为 (0,1,0,0 ,0,0,0)。当然，这为我们的特征空间增加了七个维度，并增加了七个需要训练的权重。

A quick search on the Kaggle website for data sets for regression returns many excellent data sets and related notebooks. I randomly chose a simple Fish Market data set that we will use to explain our upcoming mathematics. Our goal is to build a model that predicts the weight of a fish given its five different length measurements, or features, labeled in the data set as Length1, Length2, Length3, Height, and Width (see Figure 3-2). For the sake of simplicity, we choose not to incorporate the categorical feature, Species, into this model, even though we could (and that would give us better predictions, since a fish’s type is a good predictor of its weight). If we choose to include the Species feature, then we would have to convert its values into numerical values, using one-hot encoding, which means exactly what it sounds like: assign a code for each fish made up of ones and zeros based on its category (type). Our Species feature has seven categories: Perch, Bream, Roach, Pike, Smelt, Parkki, and Whitefish. So if our fish is a Pike, we would code its species as (0,0,0,1,0,0,0), and if it is a Bream we would code its species as (0,1,0,0,0,0,0). Of course this adds seven more dimensions to our feature space and seven more weights to train.

图 3-2。从Kaggle下载的 Fish Market 数据集的前五行。重量列是目标特征，我们的目标是建立一个模型，根据长度测量值预测新鱼的重量。

让我们节省墨水空间并将五个功能重新标记为$X_1$,$X_2$,$X_3$,$X_4$， 和$X_5$，然后将鱼的重量写为这五个特征的函数，$y=F（X_1,X_2,X_3,X_4,X_5）$。这样，一旦我们为该函数确定了可接受的公式，我们所要做的就是输入某条鱼的特征值，我们的函数将输出该鱼的预测重量。

Let’s save ink space and relabel our five features as $x_1$, $x_2$, $x_3$, $x_4$, and $x_5$, then write the fish weight as a function of these five features, $y=f(x_1,x_2,x_3,x_4,x_5)$. This way, once we settle on an acceptable formula for this function, all we have to do is input the feature values for a certain fish and our function will output the predicted weight of that fish.

本节为接下来的一切奠定了基础，因此首先了解它的组织方式很重要：

This section builds a foundation for everything to come, so it is important to first see how it is organized:

训练功能

Training function

• 参数模型与非参数模型。

• Parametric models versus nonparametric models.

损失函数

Loss function

• 预测值与真实值。

• 绝对值距离与平方距离。

• 具有奇点（尖点）的函数。

• 对于线性回归，损失函数是均方误差。

• 本书中的向量始终是列向量。

• 训练、验证和测试子集。

• 当训练数据具有高度相关的特征时。

• The predicted value versus the true value.

• The absolute value distance versus the squared distance.

• Functions with singularities (pointy points).

• For linear regression, the loss function is the mean squared error.

• Vectors in this book are always column vectors.

• The training, validation, and test subsets.

• When the training data has highly correlated features.

优化

Optimization

• 凸面景观与非凸面景观。

• 我们如何找到函数的最小化点？

• 简而言之，微积分。

• 一维优化示例。

• 我们一直使用的线性代数表达式的导数。

• 最小化均方误差损失函数。

• 注意：将大矩阵相互相乘的开销非常大，而是将矩阵与向量相乘。

• 注意：我们永远不想拟合训练数据出色地。

• Convex landscapes versus nonconvex landscapes.

• How do we locate minimizers of functions?

• Calculus in a nutshell.

• A one-dimensional optimization example.

• Derivatives of linear algebra expressions that we use all the time.

• Minimizing the mean squared error loss function.

• Caution: multiplying large matrices by each other is very expensive—multiply matrices by vectors instead.

• Caution: we never want to fit the training data too well.

培训功能

Training Function

一个快速对数据的探索，如根据各种长度特征绘制权重，使我们能够假设一个线性模型（尽管在这种情况下非线性模型可能更好）。也就是说，我们假设权重与长度特征线性相关（见图3-3）。

A quick exploration of the data, as in plotting the weight against the various length features, allows us to assume a linear model (even though a nonlinear one could be better in this case). That is, we assume that the weight depends linearly on the length features (see Figure 3-3).

这意味着鱼的重量y可以使用五个不同长度测量值的线性组合以及偏差项来计算${\omega }_0$，给出以下训练函数：

This means that the weight of a fish, y, can be computed using a linear combination of its five different length measurements, plus a bias term ${\omega }_0$, giving the following training function:

$y={\omega }_0+{\omega }_1{X}_1+{\omega }_2{X}_2+{\omega }_3{X}_3+{\omega }_4{X}_4+{\omega }_5{X}_5$

在我们在建模过程中做出使用线性训练函数的重大决定之后$F（X_1,X_2,X_3,X_4,X_5）$，我们要做的就是找到合适的参数值${\omega }_0$,${\omega }_1$,${\omega }_2$,${\omega }_3$,${\omega }_4$， 和${\omega }_5$。我们将学习最适合我们的价值观$\omega$来自数据。使用数据寻找合适的过程$\omega$称为训练模型。经过训练的模型是这样的模型：$\omega$值已经决定了。

After our major decision in the modeling process to use a linear training function $f(x_1,x_2,x_3,x_4,x_5)$, all we have to do is to find the appropriate values of the parameters ${\omega }_0$, ${\omega }_1$, ${\omega }_2$, ${\omega }_3$, ${\omega }_4$, and ${\omega }_5$. We will learn the best values for our $\omega$’s from the data. The process of using the data to find the appropriate $\omega$’s is called training the model. A trained model is then a model where the $\omega$ values have been decided on.

一般来说，训练函数，无论是线性的还是非线性的，包括那些代表神经网络的函数，都有未知的参数，$\omega$的，我们需要从给定的数据中学习。对于线性模型，每个参数在预测过程中赋予每个特征一定的权重。所以如果值${\omega }_2$大于 的值${\omega }_5$，那么假设第二个和第五个特征具有可比的尺度，那么第二个特征在我们的预测中比第五个特征发挥更重要的作用。这是在训练模型之前对数据进行缩放或标准化的好处之一。另一方面，如果该值${\omega }_3$与第三个特征相关的死亡，意义变为零或可以忽略不计，那么第三个特征可以从数据集中省略，因为它在我们的预测中不起任何作用。因此，学习我们的$\omega$数据中的 允许我们以数学方式计算每个特征对我们的预测的贡献（或者如果在训练之前的数据准备阶段组合了某些特征，则特征组合的重要性）。换句话说，模型学习数据特征如何交互以及这些交互的强度。其寓意是，通过训练有素的学习功能，我们可以量化特征如何组合在一起以产生观察到的和尚未观察到的结果。

In general, training functions, whether linear or nonlinear, including those representing neural networks, have unknown parameters, $\omega$’s, that we need to learn from the given data. For linear models, each parameter gives each feature a certain weight in the prediction process. So if the value of ${\omega }_2$ is larger than the value of ${\omega }_5$, then the second feature plays a more important role than the fifth feature in our prediction, assuming that the second and fifth features have comparable scales. This is one of the reasons it is good to scale or normalize the data before training the model. If, on the other hand, the value ${\omega }_3$ associated with the third feature dies, meaning becomes zero or negligible, then the third feature can be omitted from the data set, as it plays no role in our predictions. Therefore, learning our $\omega$’s from the data allows us to mathematically compute the contribution of each feature to our predictions (or the importance of feature combinations if some features were combined during the data preparation stage, before training). In other words, the models learn how the data features interact and how strong these interactions are. The moral is that through a trained learning function, we can quantify how features come together to produce both observed and yet-to-be observed results.

图 3-3。鱼市场数值特征的散点图。有关更多详细信息，请参阅本书的 GitHub 页面，或Kaggle 上与此数据集相关的一些公共笔记本。

损失函数

Loss Function

我们有说服自己，下一个合乎逻辑的步骤是为$\omega$使用我们拥有的数据出现在（我们的线性参数模型）训练函数中。为此，我们需要优化适当的损失函数。

We have convinced ourselves that the next logical step is finding suitable values for the $\omega$’s that appear in the training function (of our linear parametric model), using the data that we have. To do that, we need to optimize an appropriate loss function.

预测值与真实值

The predicted value versus the true value

认为我们为每个未知数分配一些随机数值${\omega }_0$,${\omega }_1$,${\omega }_2$,${\omega }_3$,${\omega }_4$， 和${\omega }_5$——比如说，${\omega }_0=-3$,${\omega }_1=4$,${\omega }_2=0。2$,${\omega }_3=0。03$,${\omega }_4=0。4$， 和${\omega }_5=0。5$。那么线性训练函数的公式$y={\omega }_0+{\omega }_1{X}_1+{\omega }_2{X}_2+{\omega }_3{X}_3+{\omega }_4{X}_4+{\omega }_5{X}_5$变成：

Suppose we assign some random numerical values for each of our unknown ${\omega }_0$, ${\omega }_1$, ${\omega }_2$, ${\omega }_3$, ${\omega }_4$, and ${\omega }_5$—say, for example, ${\omega }_0=-3$, ${\omega }_1=4$, ${\omega }_2=0.2$, ${\omega }_3=0.03$, ${\omega }_4=0.4$, and ${\omega }_5=0.5$. Then the formula for the linear training function $y={\omega }_0+{\omega }_1{x}_1+{\omega }_2{x}_2+{\omega }_3{x}_3+{\omega }_4{x}_4+{\omega }_5{x}_5$ becomes:

$y=-3+4X_1+0。2X_2+0。03X_3+0。4X_4+0。5X_5$

并准备做出预测。代入长度特征的数值${我}^{tH}$鱼，然后获得该鱼的重量的预测值。例如，我们数据集中的第一条鱼是鳊鱼，并且具有长度测量值$X_1^1=23。2$,$X_2^1=25。4$,$X_3^1=30$,$X_4^1=11。52$， 和$X_5^1=4。02$。将它们插入训练函数中，我们就可以预测这条鱼的重量：

and is ready to make predictions. Plug in numerical values for the length features of the $i^{th}$ fish, then obtain a predicted value for the weight of this fish. For example, the first fish in our data set is a bream and has length measurements $x_1^1=23.2$, $x_2^1=25.4$, $x_3^1=30$, $x_4^1=11.52$, and $x_5^1=4.02$. Plugging these into the training function, we get the prediction for the weight of this fish:

$\begin{array}{cc}\hfill y_{pred我Ct}^1& ={\omega }_0+{\omega }_1{X}_1^1+{\omega }_2{X}_2^1+{\omega }_3{X}_3^1+{\omega }_4{X}_4^1+{\omega }_5{X}_5^1\hfill \\ & =-3+4（23。2）+0。2（25。4）+0。03（30）+0。4（11。52）+0。5（4。02）\hfill \\ & =102。\mathrm{第398章}\text{克}\hfill \end{array}$

一般来说，对于${我}^{tH}$鱼，我们有：

In general, for the $i^{th}$ fish, we have:

$y_{pred我Ct}^{我}={\omega }_0+{\omega }_1{X}_1^{我}+{\omega }_2{X}_2^{我}+{\omega }_3{X}_3^{我}+{\omega }_4{X}_4^{我}+{\omega }_5{X}_5^{我}$

然而，所考虑的鱼具有一定的真实重量，$y_{tr你e}^{我}$，如果它属于标记数据集，则它是它的标签。对于我们数据集中的第一条鱼，真实重量是$y_{tr你e}^1=\mathrm{第242章}$克。我们随机选择的线性模型$\omega$预计重量为 102.398 克。这当然是相当遥远的，因为我们没有校准$\omega$根本没有价值观。无论如何，我们可以测量模型预测的权重与真实权重之间的误差，然后找到在我们的选择方面做得更好的方法这$\omega$的。

The fish under consideration, however, has a certain true weight, $y_{true}^i$, which is its label if it belongs in the labeled data set. For the first fish in our data set, the true weight is $y_{true}^1=242$ grams. Our linear model with randomly chosen $\omega$ values predicted 102.398 grams. This is of course pretty far off, since we did not calibrate the $\omega$ values at all. In any case, we can measure the error between the weight predicted by our model and the true weight, then find ways to do better in terms of our choices for the $\omega$’s.

绝对值距离与平方距离

The absolute value distance versus the squared distance

之一数学的好处在于它有多种方法可以使用不同的距离度量来测量事物之间的距离。例如，我们可以天真地测量两个量之间的距离，如果它们不同则为 1，如果相同则为 0，编码为：不同 - 1，相似 - 0。当然，使用这样一个朴素的度量，我们会丢失大量信息，因为 2 和 10 等数量之间的距离将等于 2 和 100 万之间的距离，即 1。

One of the nice things about mathematics is that it has multiple ways to measure how far things are from each other, using different distance metrics. For example, we can naively measure the distance between two quantities as being 1 if they are different, and 0 if they are the same, encoding the words: different-1, similar-0. Of course, using such a naive metric, we lose a ton of information, since the distance between quantities such as 2 and 10 will be equal to the distance between 2 and 1 million, namely 1.

有一些在机器学习中很流行的距离度量。我们先介绍一下最常用的两个：

There are some distance metrics that are popular in machine learning. We first introduce the two most commonly used:

• 绝对值距离：$|y_{pred我Ct}-y_{tr你e}|$，源于微积分函数$\left|X\right|$。

• The absolute value distance: $|y_{predict}-y_{true}|$, stemming from the calculus function $\left|x\right|$.

• 距离平方：$|y_{pred我Ct}-y_{tr你e}{|}^2$，源于微积分函数${\left|X\right|}^2$（这与$X^2$对于标量）。当然，这也会对单位进行平方。

• The squared distance: $|y_{predict}-y_{true}{|}^2$, stemming from the calculus function ${\left|x\right|}^2$ (which is the same as $x^2$ for scalar quantities). Of course, this will square the units as well.

检查函数的图形$\left|X\right|$和$X^2$在图 3-4中，我们注意到 (0,0) 点函数平滑度存在很大差异。功能$\left|X\right|$在该点有一个角，使其在 x = 0 处不可微。$\left|X\right|$x = 0 时，许多实践者（和数学家！）不再将这个函数或具有类似奇点的函数合并到他们的模型中。但是，让我们将以下内容铭刻在我们的大脑中。

Inspecting the graphs of the functions $\left|x\right|$ and $x^2$ in Figure 3-4, we notice a great difference in function smoothness at the point (0,0). The function $\left|x\right|$ has a corner at that point, rendering it undifferentiable at x = 0. This singularity of $\left|x\right|$ at x = 0 turns many practitioners (and mathematicians!) away from incorporating this function, or functions with similar singularities, into their models. However, let’s engrave the following into our brains.

数学模型是灵活的。当我们遇到障碍时，我们会更深入地挖掘，了解发生了什么，然后克服障碍。

Mathematical models are flexible. When we encounter a hurdle we dig deeper, understand what’s going on, then we work around the hurdle.

图 3-4。左：图表$\left|X\right|$有一个角在$X=0$，此时其导数未定义。右：图表${\left|X\right|}^2$平滑于$X=0$，所以它的导数在那里没有问题。

除了函数的规律性不同之外$\left|X\right|$和${\left|X\right|}^2$（意味着它们是否在所有点上都有导数），在决定将任一函数纳入我们的误差公式之前，我们还需要注意一点：如果一个数字很大，那么它的平方更大。这个简单的观察意味着，如果我们决定使用真实值和预测值之间的平方距离来测量误差，那么我们的方法将对数据中的异常值更加敏感。一个混乱的异常值可能会使我们的整个预测函数偏向它，从而偏离数据中更普遍的模式。理想情况下，在将数据输入任何机器学习模型之前，我们会在数据准备步骤中处理异常值并决定是否应该保留它们。

Other than the difference in the regularity of the functions $\left|x\right|$ and ${\left|x\right|}^2$ (meaning whether they have derivatives at all points or not), there is one more point that we need to pay attention to before deciding to incorporate either function into our error formula: if a number is large, then its square is even larger. This simple observation means that if we decide to measure the error using squared distances between true values and predicted values, then our method will be more sensitive to the outliers in the data. One messed-up outlier might skew our whole prediction function toward it, and hence away from the more prevalent patterns in the data. Ideally, we would have taken care of outliers and decided whether we should keep them or not during the data preparation step, before feeding the data into any machine learning model.

最后一个区别$\left|X\right|$（以及类似的分段线性函数）和$X^2$（以及类似的非线性但可微函数）是$\left|X\right|$很简单：

One last difference between $\left|x\right|$ (and similar piecewise linear functions) and $x^2$ (and similar nonlinear but differentiable functions) is that the derivative of $\left|x\right|$ is very easy:

• 如果x > 0，则为 1；如果x < 0 ，则为 –1 （如果x = 0，则未定义）
• 1 if x > 0, –1 if x < 0 (and undefined if x = 0)

在涉及数十亿计算步骤的模型中，在使用导数时无需评估任何内容的属性$\left|X\right|$事实证明是非常有价值的。既不是线性也不是分段线性的函数的导数通常涉及求值（因为它们的公式中也有x，而不仅仅是常数，就像分段线性情况一样），这在以下方面可能会很昂贵：大数据设置。

In a model that involves billions of computational steps, this property where there is no need to evaluate anything when using the derivative of $\left|x\right|$ proves to be extremely valuable. Derivatives of functions that are neither linear nor piecewise linear usually involve evaluations (because they also have x’s in their formulas and not only constants, like in the piecewise linear case), which can be expensive in big data settings.

具有奇点的函数

Functions with singularities

一般来说，可微函数图没有尖点、扭结、拐角或任何尖锐的东西。如果它们确实具有这样的奇点，那么函数在这些点上没有导数。原因是，在某个点处，您可以在函数图像上绘制两条不同的切线，具体取决于您决定将切线绘制到该点的左侧还是右侧（见图3-5） 。回想一下，函数在某一点的导数是函数图形在该点的切线的斜率。如果有两个不同的斜率，那么我们无法定义该点的导数。

In general, graphs of differentiable functions do not have cusps, kinks, corners, or anything pointy. If they do have such singularities, then the function has no derivative at these points. The reason is that at a pointy point, you can draw two different tangent lines to the graph of the function, depending on whether you decide to draw the tangent line to the left or to the right of the point (see Figure 3-5). Recall that the derivative of a function at a point is the slope of the tangent line to the graph of the function at that point. If there are two different slopes, then we cannot define the derivative at the point.

图 3-5。在奇点处，导数不存在。在这些点处存在不止一种可能的切线斜率。

切线斜率的这种不连续性给依赖于计算函数导数的方法（例如梯度下降法）带来了问题。这里的问题是双重的：

This discontinuity in the slope of the tangent creates a problem for methods that rely on evaluating the derivative of the function, such as the gradient descent method. The problem here is twofold:

未定义导数

Undefined derivative

您应该使用什么衍生价值？如果您碰巧落在一个奇怪的尖点上，那么该方法不知道该怎么做，因为那里没有定义的导数。有些人为该点的导数指定一个值（称为次梯度或次微分）并继续。事实上，我们不幸恰好落在那个可怕的时刻的可能性有多大？除非函数的地形看起来像阿尔卑斯山的崎岖地形（实际上，很多都是如此），否则数值方法可能会设法避免它们。

What derivative value should you use? If you happen to land at a quirky pointy point, then the method doesn’t know what to do, since there is no defined derivative there. Some people assign a value for the derivative at that point (called the subgradient or the subdifferential) and move on. In reality, what are the odds that we will be unlucky enough to land exactly at that one horrible point? Unless the landscape of the function looks like the rough terrain of the Alps (actually, many do), the numerical method might manage to avoid them.

不稳定

Instability

另一个问题是不稳定。由于当您穿过该点的函数范围时，导数的值会突然跳跃，因此使用该导数的方法也会突然改变值，如果您试图在某个地方收敛，则会产生不稳定。想象一下，您正在图 3-6中的瑞士阿尔卑斯山（损失函数的景观）中徒步旅行，您的目的地是您可以在山谷中看到的那个漂亮的小镇（误差值最低的地方）。然后突然你被某个外星人（外星人就是依靠这种突然变化的导数的数学搜索方法）带到了山的另一边，在那里你再也看不到你的目的地了。事实上，现在你在山谷中只能看到一些丑陋的灌木丛和一个极其狭窄的峡谷，如果你的方法将你带到那里，它可能会困住你。你与最初目的地的融合现在即使没有完全迷失，也不稳定。

图 3-6。瑞士阿尔卑斯山：优化类似于徒步旅行功能景观

The other problem is instability. Since the value of the derivative jumps so abruptly as you traverse the landscape of the function across this point, a method using this derivative will abruptly change value as well, creating instabilities if you are trying to converge somewhere. Imagine you are hiking down the Swiss Alps in Figure 3-6 (the landscape of the loss function) and your destination is that pretty little town that you can see down in the valley (the place with the lowest error value). Then suddenly you get carried away by some alien (the alien is the mathematical search method relying on this abruptly changing derivative) to the other side of the mountain, where you cannot see your destination anymore. In fact, now all you can see in the valley are some ugly shrubs and an extremely narrow canyon that can trap you if your method carries you there. Your convergence to your original destination is now unstable, if not totally lost.

Figure 3-6. Swiss Alps: optimization is similar to hiking the landscape of a function

然而，具有这种奇点的函数在机器学习中一直被使用。我们会在一些神经网络训练函数的公式（修正线性单位函数——谁命名这些函数？）、一些损失函数（绝对值距离）和一些正则化术语中遇到它们（套索回归——谁也给这些命名了？）。

Nevertheless, functions with such singularities are used all the time in machine learning. We will encounter them in the formulas of some neural network training functions (rectified linear unit function—who names these?), in some loss functions (absolute value distance), and in some regularizing terms (lasso regression—who names these too?).

对于线性回归，损失函数是均方误差

For linear regression, the loss function is the mean squared error

回到本节的主要目标：构建一个误差函数，也称为损失函数，它对我们的模型在进行预测时犯下的误差进行编码，并且必须使其很小。

Back to the main goal for this section: constructing an error function, also called the loss function, which encodes how much error our model commits when making its predictions, and must be made small.

对于线性回归，我们使用均方误差函数。该函数对m 个数据点的预测值与真实值之间的平方距离误差进行平均（我们将很快提到此处要包括哪些数据点）：

For linear regression, we use the mean squared error function. This function averages over the squared distance errors between the prediction and the true value for m data points (we will mention which data points to include here shortly):

$\text{意思是}\text{平方}\text{错误}=\frac{1}{米}\left(|y_{pred我Ct}^1-y_{tr你e}^1{|}^2+|y_{pred我Ct}^2-y_{tr你e}^2{|}^2+\cdots +{|y_{pred我Ct}^{米}-y_{tr你e}^{米}|}^2\right)$

让我们使用求和符号将上面的表达式写得更紧凑：

Let’s write the above expression more compactly using the sum notation:

$\text{意思是}\text{平方}\text{错误}=\frac{1}{米}{\Sigma }_{我=1}^{米}{|y_{pred我Ct}^{我}-y_{tr你e}^{我}|}^2$

现在我们养成了使用向量和矩阵的更紧凑的线性代数表示法的好习惯。事实证明，这个习惯在现场非常方便，因为我们不想在尝试跟踪索引时被淹没。指数可能会潜入我们了解一切的美好梦想，并迅速将其转变为非常可怕的噩梦。使用紧凑线性代数表示法的另一个重要原因是，为机器学习模型构建的软件和硬件都针对矩阵和张量进行了优化（想象一下由分层矩阵组成的对象，例如三维盒子而不是平面正方形） ) 计算。此外，数值线性代数这个美丽的领域已经解决了许多潜在的问题，并为我们享受执行各种矩阵计算的快速方法铺平了道路。

Now we get into the great habit of using the even more compact linear algebra notation of vectors and matrices. This habit proves extremely handy in the field, as we don’t want to drown while trying to keep track of indices. Indices can sneak up into our rosy dreams of understanding everything and quickly transform them into very scary nightmares. Another important reason to use the compact linear algebra notation is that both the software and the hardware built for machine learning models are optimized for matrix and tensor (think of an object made of layered matrices, like a three-dimensional box instead of a flat square) computations. Moreover, the beautiful field of numerical linear algebra has worked through many potential problems and paved the way for us to enjoy the fast methods to perform all kinds of matrix computations.

使用线性代数符号，我们可以将均方误差写为：

Using linear algebra notation, we can write the mean squared error as:

$\text{意思是}\text{平方}\text{错误}=\frac{1}{米}{（{\overrightarrow{y}}_{pred我Ct}-{\overrightarrow{y}}_{tr你e}）}^t（{\overrightarrow{y}}_{pred我Ct}-{\overrightarrow{y}}_{tr你e}）=\frac{1}{米}{\parallel {\overrightarrow{y}}_{pred我Ct}-{\overrightarrow{y}}_{tr你e}\parallel }_{{我}^2}^2$

最后一个等式引入了${我}^2$ 向量的范数，根据定义，向量的范数就是$\sqrt{\text{和}\text{的}\text{正方形}\text{的}\text{它是}\text{成分}}$。

The last equality introduces the $l^2$ norm of a vector, which by definition is just the $\sqrt{\text{sum}\text{of}\text{squares}\text{of}\text{its}\text{components}}$.

主要想法：我们构建的损失函数对涉及训练过程的数据点的预测与真实情况之间的差异进行编码，以某种规范（充当距离的数学实体）进行测量。我们还可以使用许多其他规范，但是${我}^2$ 标准很漂亮受欢迎的。

Take-home idea: the loss function that we constructed encodes the difference between the predictions and the ground truths for the data points involved the training process, measured in some norm—a mathematical entity that acts as a distance. There are many other norms that we could have used, but the $l^2$ norm is pretty popular.

符号：本书中的向量始终是列向量

Notation: Vectors in this book are always column vectors

成为整本书中的符号一致，所有向量都是列向量。所以如果一个向量$\overrightarrow{v}$有四个组成部分，符号$\overrightarrow{v}$代表$\left(\begin{array}{c}{v}_1\\ v_2\\ v_3\\ v_4\end{array}\right)$。

To be consistent in notation throughout the book, all vectors are column vectors. So if a vector $\overrightarrow{v}$ has four components, the symbol $\overrightarrow{v}$ stands for $\left(\begin{array}{c}{v}_1\\ v_2\\ v_3\\ v_4\end{array}\right)$.

向量的转置$\overrightarrow{v}$那么，总是一个行向量。上述具有四个分量的向量的转置为${\overrightarrow{v}}^t=\left(\begin{array}{cccc}{v}_1& v_2& v_3& v_4\end{array}\right)$。

The transpose of a vector $\overrightarrow{v}$ is, then, always a row vector. The transpose of the above vector with four components is ${\overrightarrow{v}}^t=\left(\begin{array}{cccc}{v}_1& v_2& v_3& v_4\end{array}\right)$.

我们还可以使用点积表示法（也称为标量积，因为我们将两个向量相乘，但我们的答案是标量数）。两个向量的点积$\overrightarrow{A}。\overrightarrow{乙}$是一样的事情${\overrightarrow{A}}^t\overrightarrow{乙}$。在本质上，${\overrightarrow{A}}^t\overrightarrow{乙}$将列向量视为形状矩阵，向量长度乘以 1，并将其转置视为形状矩阵，向量长度乘以 1。

We might also use the dot product notation (also called the scalar product because we multiply two vectors but our answer is a scalar number). The dot product of two vectors $\overrightarrow{a}.\overrightarrow{b}$ is the same thing as ${\overrightarrow{a}}^t\overrightarrow{b}$. In essence, ${\overrightarrow{a}}^t\overrightarrow{b}$ thinks of a column vector as a matrix of shape, length of the vector by 1, and its transpose as a matrix of shape, 1 by length of the vector.

现在假设$\overrightarrow{A}$和$\overrightarrow{乙}$有四个组成部分，那么：

Suppose now that $\overrightarrow{a}$ and $\overrightarrow{b}$ have four components, then:

${\overrightarrow{A}}^t\overrightarrow{乙}=\left(\begin{array}{cccc}{A}_1& A_2& A_3& A_4\end{array}\right)\left(\begin{array}{c}{乙}_1\\ {乙}_2\\ {乙}_3\\ {乙}_4\end{array}\right)=A_1{乙}_1+A_2{乙}_2+A_3{乙}_3+A_4{乙}_4={\Sigma }_{我=1}^4{A}_{我}{乙}_{我}$

而且，

Moreover,

$\parallel \overrightarrow{A}{\parallel }_{{我}^2}^2={\overrightarrow{A}}^t\overrightarrow{A}=A_1^2+A_2^2+A_3^2+A_4^2$

相似地，

Similarly,

$\parallel \overrightarrow{乙}{\parallel }_{{我}^2}^2={\overrightarrow{乙}}^t\overrightarrow{乙}={乙}_1^2+{乙}_2^2+{乙}_3^2+{乙}_4^2$

这样，我们自始至终都使用矩阵表示法，并且只在字母上方放置一个箭头来指示我们正在处理列向量。

This way, we use matrix notation throughout, and we only put an arrow above a letter to indicate that we are dealing with a column vector.

训练、验证和测试子集

The training, validation, and test subsets

哪个我们的损失函数中包含数据点吗？我们是否包含整个数据集、一小批数据，甚至只包含一个点？我们是否测量训练子集、验证子集或测试子集中的数据点的均方误差？这些子集到底是什么？

Which data points do we include in our loss function? Do we include the whole data set, some small batches of it, or even only one point? Are we measuring this mean squared error for the data points in the training subset, the validation subset, or the test subset? And what are these subsets anyway?

在实践中，我们将数据集分为三个子集：

In practice, we split a data set into three subsets:

训练子集

Training subset

这是我们用来拟合训练函数的数据子集。这意味着该子集中的数据点将被纳入我们的损失函数中（通过将其特征值和标签插入到损失函数中）$y_{pred我Ct}$和$y_{tr你e}$损失函数）。

This is the subset of the data that we use to fit our training function. This means that the data points in this subset are the ones that get incorporated into our loss function (by plugging their feature values and label into the $y_{predict}$ and the $y_{true}$ of the loss function).

验证子集

Validation subset

这该子集中的数据点有多种使用方式：

• 常见的描述是我们使用这个子集来调整机器学习模型的超参数。这超参数是机器学习模型中不属于机器学习模型的任何参数。$\omega$我们正在尝试解决的训练函数的问题。在机器学习中，有很多这样的东西，它们的值会影响模型的结果和性能。超参数的例子包括（你不必知道它们是什么）：梯度下降法中出现的学习率；支持向量机方法中决定边距宽度的超参数；原始数据分为训练、验证和测试子集的百分比；进行随机批量梯度下降时的批量大小；权重衰减超参数，例如岭回归、套索回归和弹性网络回归中使用的超参数；动量方法附带的超参数，例如动量梯度下降和 ADAM（这些项具有加速方法收敛到最小值的项，并且这些项与需要在测试和部署之前调整的超参数相乘）；号码优化过程中的历元（优化器已经看到的整个训练子集的传递次数）；以及神经网络的架构（例如层数、每层的宽度等）。

• 验证子集还可以帮助我们知道在过度拟合训练子集之前何时停止优化。

• 它还可以作为测试集来比较不同机器学习模型在同一数据集上的性能，例如比较线性回归模型、随机森林和神经网络的性能。

The data points in this subset are used in multiple ways:

• The common description is that we use this subset to tune the hyperparameters of the machine learning model. The hyperparameters are any parameters in the machine learning model that are not the $\omega$’s of the training function that we are trying to solve for. In machine learning, there are many of these, and their values affect the results and the performance of the model. Examples of hyperparameters include (you don’t have to know what these are yet): the learning rate that appears in the gradient descent method; the hyperparameter that determines the width of the margin in support vector machine methods; the percentage of original data split into training, validation, and test subsets; the batch size when doing randomized batch gradient descent; weight decay hyperparameters such as those used in ridge, lasso, and elastic net regression; hyperparameters that come with momentum methods such as gradient descent with momentum and ADAM (these have terms that accelerate the convergence of the method toward the minimum, and these terms are multiplied by hyperparameters that need to be tuned before testing and deployment); the number of epochs during the optimization process (the number of passes over the entire training subset that the optimizer has seen); and the architecture of a neural network (such as the number of layers, the width of each layer, etc.).

• The validation subset also helps us know when to stop optimizing before overfitting our training subset.

• It also serves as a test set to compare the performance of different machine learning models on the same data set, for example, comparing the performance of a linear regression model to a random forest to a neural network.

测试子集

Test subset

后在决定使用的最佳模型（或平均或聚合多个模型的结果）并训练模型后，我们使用这个未受影响的数据子集作为模型部署到现实世界之前的最后阶段测试。由于该模型之前没有见过该子集中的任何数据点（这意味着它在优化过程中没有包含任何数据点），因此它可以被认为是最接近真实情况的模拟。这使我们能够在开始完全应用模型之前判断模型的性能新的现实世界数据。

After deciding on the best model to use (or averaging or aggregating the results of multiple models) and training the model, we use this untouched subset of the data as a last-stage test for our model before deployment into the real world. Since the model has not seen any of the data points in this subset before (which means it has not included any of them in the optimization process), it can be considered as the closest analog to a real-world situation. This allows us to judge the performance of our model before we start applying it to completely new real-world data.

回顾

Recap

在继续之前让我们回顾一下：

Let’s recap a little before moving forward:

• 我们当前的机器学习模型称为线性回归。

• Our current machine learning model is called linear regression.

• 我们的训练函数与以下公式呈线性关系：

  $y={\omega }_0+{\omega }_1{X}_1+{\omega }_2{X}_2+{\omega }_3{X}_3+{\omega }_4{X}_4+{\omega }_5{X}_5$

  这$X$是特征，并且$\omega$是未知的权重或参数。

• Our training function is linear with the formula:

  $y={\omega }_0+{\omega }_1{x}_1+{\omega }_2{x}_2+{\omega }_3{x}_3+{\omega }_4{x}_4+{\omega }_5{x}_5$

  The $x$’s are the features, and the $\omega$’s are the unknown weights or parameters.

• 如果我们将特定数据点（例如第十个数据点）的特征值代入训练函数的公式中，我们就会得到模型对该点的预测：

  $y_{pred我Ct}^{10}={\omega }_0+{\omega }_1{X}_1^{10}+{\omega }_2{X}_2^{10}+{\omega }_3{X}_3^{10}+{\omega }_4{X}_4^{10}+{\omega }_5{X}_5^{10}$

  上标10表示这些是对应于第十个数据点的值。

• If we plug in the feature values of a particular data point–for example, the tenth data point–into the formula of the training function, we get our model’s prediction for this point:

  $y_{predict}^{10}={\omega }_0+{\omega }_1{x}_1^{10}+{\omega }_2{x}_2^{10}+{\omega }_3{x}_3^{10}+{\omega }_4{x}_4^{10}+{\omega }_5{x}_5^{10}$

  The superscript 10 indicates that these are values corresponding to the tenth data point.

• 我们的损失函数是均方误差函数，其公式为：

  $\text{意思是}\text{平方}\text{错误}=\frac{1}{米}{（{\overrightarrow{y}}_{pred我Ct}-{\overrightarrow{y}}_{tr你e}）}^t（{\overrightarrow{y}}_{pred我Ct}-{\overrightarrow{y}}_{tr你e}）=\frac{1}{米}{\parallel {\overrightarrow{y}}_{pred我Ct}-{\overrightarrow{y}}_{tr你e}\parallel }_{{我}^2}^2$

• Our loss function is the mean squared error function with the formula:

  $\text{Mean}\text{Squared}\text{Error}=\frac{1}{m}{({\overrightarrow{y}}_{predict}-{\overrightarrow{y}}_{true})}^t({\overrightarrow{y}}_{predict}-{\overrightarrow{y}}_{true})=\frac{1}{m}{\parallel {\overrightarrow{y}}_{predict}-{\overrightarrow{y}}_{true}\parallel }_{l^2}^2$

• 我们想要找到的值$\omega$是最小化这个损失函数。因此下一步必须解决最小化（优化）问题。

• We want to find the values of the $\omega$’s that minimize this loss function. So the next step must be solving a minimization (optimization) problem.

为了使我们的优化过程变得更加容易，我们将再次使用线性代数（向量和矩阵）的方便表示法。这使我们能够将数据的整个训练子集作为矩阵包含在损失函数的公式中，并立即对训练子集进行计算，而不是分别对每个数据点进行计算。这个小小的符号操作使我们避免了许多错误、痛苦和繁琐的计算，其中许多组件很难在非常大的数据集上跟踪。

To make our optimization life much easier, we will once again employ the convenient notation of linear algebra (vectors and matrices). This allows us to include the entire training subset of the data as a matrix in the formula of the loss function, and do our computations immediately on the training subset, as opposed to computing on each data point separately. This little notation maneuver saves us from a lot of mistakes, pain, and tedious calculations with many components that are difficult to keep track of on very large data sets.

首先，写出我们的模型对应于训练子集的每个数据点的预测：

First, write the prediction of our model corresponding to each data point of the training subset:

$\begin{array}{cc}\hfill y_{pred我Ct}^1& =1{\omega }_0+{\omega }_1{X}_1^1+{\omega }_2{X}_2^1+{\omega }_3{X}_3^1+{\omega }_4{X}_4^1+{\omega }_5{X}_5^1\hfill \\ \hfill y_{pred我Ct}^2& =1{\omega }_0+{\omega }_1{X}_1^2+{\omega }_2{X}_2^2+{\omega }_3{X}_3^2+{\omega }_4{X}_4^2+{\omega }_5{X}_5^2\hfill \\ \hfill ⋮\\ \hfill y_{pred我Ct}^{米}& =1{\omega }_0+{\omega }_1{X}_1^{米}+{\omega }_2{X}_2^{米}+{\omega }_3{X}_3^{米}+{\omega }_4{X}_4^{米}+{\omega }_5{X}_5^{米}\hfill \end{array}$

我们可以轻松地将这个系统安排为：

We can easily arrange this system as:

$\left(\begin{array}{c}{y}_{pred我Ct}^1\\ y_{pred我Ct}^2\\ ⋮\\ y_{pred我Ct}^{米}\end{array}\right)=\left(\begin{array}{c}1\\ 1\\ ⋮\\ 1\end{array}\right){\omega }_0+\left(\begin{array}{c}{X}_1^1\\ X_1^2\\ ⋮\\ X_1^{米}\end{array}\right){\omega }_1+\left(\begin{array}{c}{X}_1^1\\ X_2^2\\ ⋮\\ X_2^{米}\end{array}\right){\omega }_2+\left(\begin{array}{c}{X}_3^1\\ X_3^2\\ ⋮\\ X_3^{米}\end{array}\right){\omega }_3+\left(\begin{array}{c}{X}_4^1\\ X_4^2\\ ⋮\\ X_4^{米}\end{array}\right){\omega }_4+\left(\begin{array}{c}{X}_5^1\\ X_5^2\\ ⋮\\ X_5^{米}\end{array}\right){\omega }_5$

或者甚至更好：

Or even better:

$\left(\begin{array}{c}{y}_{pred我Ct}^1\\ y_{pred我Ct}^2\\ ⋮\\ y_{pred我Ct}^{米}\end{array}\right)=\left(\begin{array}{cccccc}1& X_1^1& X_2^1& X_3^1& X_4^1& X_5^1\\ 1& X_1^2& X_2^2& X_3^2& X_4^2& X_5^2\\ ⋮\\ 1& X_1^{米}& X_2^{米}& X_3^{米}& X_4^{米}& X_5^{米}\end{array}\right)\left(\begin{array}{c}{\omega }_0\\ {\omega }_1\\ {\omega }_2\\ {\omega }_3\\ {\omega }_4\\ {\omega }_5\end{array}\right)$

该方程左侧的向量是${\overrightarrow{y}}_{pred我Ct}$，右侧的矩阵是用 1 向量增广的训练子集X，右侧的最后一个向量将所有未知权重整齐地装入其中。将此向量称为$\overrightarrow{\omega }$， 然后写${\overrightarrow{y}}_{pred我Ct}$就训练子集而言紧凑并且$\overrightarrow{\omega }$作为：

The vector on the lefthand side of that equation is ${\overrightarrow{y}}_{predict}$, the matrix on the righthand side is the training subset X augmented with the vector of ones, and the last vector on the righthand side has all the unknown weights packed neatly into it. Call this vector $\overrightarrow{\omega }$, then write ${\overrightarrow{y}}_{predict}$ compactly in terms of the training subset and $\overrightarrow{\omega }$ as:

${\overrightarrow{y}}_{pred我Ct}=X\overrightarrow{\omega }$

现在是均方误差损失函数的公式，我们之前写成：

Now the formula of the mean squared error loss function, which we wrote before as:

$\text{意思是}\text{平方}\text{错误}=\frac{1}{米}{（{\overrightarrow{y}}_{pred我Ct}-{\overrightarrow{y}}_{tr你e}）}^t（{\overrightarrow{y}}_{pred我Ct}-{\overrightarrow{y}}_{tr你e}）=\frac{1}{米}{\parallel {\overrightarrow{y}}_{pred我Ct}-{\overrightarrow{y}}_{tr你e}\parallel }_{{我}^2}^2$

变成：

becomes:

$\text{意思是}\text{平方}\text{错误}=\frac{1}{米}{\left(X\overrightarrow{\omega }-{\overrightarrow{y}}_{tr你e}\right)}^t\left(X\overrightarrow{\omega }-{\overrightarrow{y}}_{tr你e}\right)=\frac{1}{米}{\parallel X\overrightarrow{\omega }-{\overrightarrow{y}}_{tr你e}\parallel }_{{我}^2}^2$

我们现在准备好寻找$\overrightarrow{\omega }$最大限度地减少整洁的损失函数。为此，我们必须参观丰富而美丽的优化数学领域。

We are now ready to find the $\overrightarrow{\omega }$ that minimizes the neatly written loss function. For that, we have to visit the rich and beautiful mathematical field of optimization.

当训练数据具有高度相关的特征时

When the training data has highly correlated features

检验训练矩阵（用 1 向量增强）：

Inspecting the training matrix (augmented with the vector of ones):

$X=\left(\begin{array}{cccccc}1& X_1^1& X_2^1& X_3^1& X_4^1& X_5^1\\ 1& X_1^2& X_2^2& X_3^2& X_4^2& X_5^2\\ ⋮\\ 1& X_1^{米}& X_2^{米}& X_3^{米}& X_4^{米}& X_5^{米}\end{array}\right)$

出现在向量中${\overrightarrow{y}}_{pred我Ct}=X\overrightarrow{\omega }$，均方误差损失函数的公式，以及后面确定未知数的公式$\overrightarrow{\omega }$（也称为正规方程）：

that appears in the vector ${\overrightarrow{y}}_{predict}=X\overrightarrow{\omega }$, the formula of the mean squared error loss function, and later the formula that determines the unknown $\overrightarrow{\omega }$ (also called the normal equation):

$\overrightarrow{\omega }={（X^{t}X）}^{-1}{X}^t{\overrightarrow{y}}_{tr你e}$

我们可以看到，如果数据的两个或多个特征（ x列）高度相关，我们的模型可能会出现问题。这意味着特征之间存在很强的线性关系，因此可以使用其他特征的线性组合来确定（或接近确定）其中一个特征。因此，相应的特征列不是线性独立的（或接近非线性独立的）。对于矩阵来说，这是一个问题，因为它表明矩阵要么不能逆，要么是病态的。病态矩阵会在计算中产生很大的不稳定性，因为训练数据的微小变化（必须假设）会导致模型参数发生很大的变化，从而导致其预测不可靠。

we can see how our model might have a problem if two or more features (x columns) of the data are highly correlated. This means that there is a strong linear relationship between the features, so one of these features can be determined (or nearly determined) using a linear combination of the others. Thus, the corresponding feature columns are not linearly independent (or close to not being linearly independent). For matrices, this is a problem, since it indicates that the matrix either cannot be inverted or is ill conditioned. Ill-conditioned matrices produce large instabilities in computations, since slight variations in the training data (which must be assumed) produce large variations in the model parameters and hence render its predictions unreliable.

我们希望在计算中得到良条件矩阵，因此我们必须消除不良条件的来源。当我们具有高度相关的特征时，一种可能的途径是在我们的模型中仅包含其中之一，因为其他特征不会添加太多信息。另一种解决方案是应用降维技术，例如主成分分析，我们将在第 11 章中遇到。Fish Market 数据集具有高度相关的特征，随附的 Jupyter Notebook 解决了这些问题。

We desire well-conditioned matrices in our computations, so we must get rid of the sources of ill conditioning. When we have highly correlated features, one possible avenue is to include only one of them in our model, as the others do not add much information. Another solution is to apply dimension reduction techniques such as principal component analysis, which we will encounter in Chapter 11. The Fish Market data set has highly correlated features, and the accompanying Jupyter Notebook addresses those.

也就是说，值得注意的是，一些机器学习模型，例如决策树和随机森林（即将讨论）不受相关特征的影响，而其他模型，例如当前的线性回归模型以及下一个逻辑回归和支持向量机模型，受到它们的负面影响。对于神经网络模型来说，尽管它们可以在训练过程中学习数据特征中涉及的相关性，但如果提前处理好这些冗余，除了节省计算量之外，它们的性能也会更好成本和时间。

That said, it is important to note that some machine learning models, such as decision trees and random forests (discussed soon) are not affected by correlated featured, while others, such as the current linear regression model, and the next logistic regression and support vector machine models, are negatively affected by them. As for neural network models, even though they can learn the correlations involved in the data features during training, they perform better when these redundancies are taken care of ahead of time, in addition to saving computation cost and time.